1996 CARL Research Award: Final Report

Defining and Measuring the Teaching-Learning
Library's Impact on Institutional Outcomes

Bonnie Gratch Lindauer

THE MARCH 1997 issue of the CARL Newsletter contained a mid-project status report, which summarized the findings related to academic libraries' and librarians' roles as described in regional accreditation and ACRL professional standards. The following report highlights some of the findings from the literature of higher education outcomes assessment and from academic library performance evaluation studies. It also provides the author's suggestions and ideas for defining and measuring the library's impact on those outcomes that are valued by academic institutions. The regional accreditation and ACRL sectional standards were used to identify the current expectations for academic libraries' contributions to educational outcomes. In addition, I found very useful several higher education research reports on undergraduate student outcomes, literature describing new and emerging roles for academic libraries, information literacy competencies and objectives, and academic library multiple stakeholder research findings.1 Before presenting some of the major findings and suggestions, I want to make explicit several assumptions and share a few definitions so that my use of terminology is clear. Assumptions and Definitions I made several assumptions which guided my work. First, I believe that using empirical data derived from multiple stakeholders research and national standards and guidelines, it is possible to identify some of the key indicators of effective/quality academic libraries. This article presents selected indicators and criteria that could be considered key for most academic libraries. An article containing a full report is forthcoming, and interested readers are encouraged to contact me for information about when it will be available. Second, while I realize that further empirical research on the impact of academic libraries on valued educational outcomes is needed, this article advocates approaches to measuring and assessing impact that make use of a variety of field-tested quantitative and qualitative measures. Hence a multi-methods and ongoing, longitudinal assessment effort is promulgated, one that reflects the advice of virtually all writers who emphasize that assessment should not be seen as a major undertaking once in awhile, but rather integrated into on-going data collection and planning activities. Third, I have been influenced by many points made in the literature about the future and emerging roles of largely digital academic libraries, particularly since nearly every writer on this topic stresses the importance of an active teaching-learning role, not just in support of the various academic curricula, but also the role of teaching information literacy as a major contribution to academic excellence. Other roles that may need to be reflected in performance criteria and indicators reflect such changing roles and functions as: creation and support of "holistic computing environments" that make the technology work for all users; delivery of around the clock reference and instructional services over the network; partnering across administrative lines for the improvement of services and resources; provision of improved electronic integrated library systems that emphasize direct-user access to both full-text and bibliographic resources; high quality document delivery service; instructional design and production of teaching materials; and creation of new knowledge packages and new access tools.2 Since the project's perspective is organizational-level assessment, library sub-organizational approaches have been excluded. Also excluded are criteria and indicators dealing with cost and cost-efficiency/effectiveness. I acknowledge the need for these measures, particularly given the increasingly high costs for maintaining quality networked services, and refer readers to McCLure and Lopata for network efficiency indicators and Kantor and Lancaster for guidance and methods evaluating traditional library sub-organizational services and resources.3 What's the difference between evaluation, assessment, measurement? What is meant by "valued institutional outcomes?" Is the evaluation of library effectiveness the same thing as library quality or performance? I use "library effectiveness" synonomously with "library quality," as well as use "evaluation" interchangeably with "assessment." The distinction between measurement and evaluation is that evaluation occurs when judgments are applied, or as defined by McClure,". . .evaluation is a systematic process which assesses the efficiency and effectiveness of certain activities and offers a judgment of their value in terms of some criteria, such as goals and objectives."4 While some writers have defined library effectiveness/quality as a broader concept than library performance, others, perhaps using a planning and management perspective, have interpreted all aspects of library operations, from its necessary infrastructure types of resources (e.g. staff and budget) to its services and products, to be incorporated by the phrase "performance measures." I have adopted McClure and Lopata's explanation: "Performance measures are a broad, managerial tool that encompass measurement of inputs (indicators of the resources essential to provide a service); outputs (indicators of the services resulting from the use of those resources); and impacts (the effects of these outputs on other variables or factors) . . . ."5 The phrases "performance criteria," "performance indicators" or "performance factors" are sometimes used interchangeably in the literature. In order to clarify certain distinctions, I use "criteria" to mean guidelines or norms, operationally employed as the basis for making a judgment or decision. Throughout this article, the use of the word "outcomes" will be reserved for expected or assumed accomplishment of institutional goals valued by the various stakeholder groups (e.g. students, faculty, administrators, librarians, other academic staff, trustees, etc.) and the word "impact(s)" will be used for those direct effects that the library has on institutional outcomes, or if more indirect, those enabling outputs it contributes to these outcomes. The use of the word "library" is understood broadly to include learning resource centers, instructional media centers, and jointly administered academic computer centers and networks. Valued Institutional Outcomes and Outputs To Which Libraries Directly Contribute Generally, academic libraries/librarians face two problems when trying to describe the impact of their services and resources on institutional outcomes: first, they often don't organize their data and other supporting documentation in ways that are accessible or meaningful to academic administrators and accreditation teams, nor do they use the language readily understood by these key stakeholders; second, they are not strategic about which measures to use for evidence or support of how their contributions effect institutional outcomes. Typically, all sorts of data are presented in annual reports and self studies, which while seemingly important do not explicitly address how the library's resources make a difference to institutional goals and outcomes. In the past ten years there have been various initiatives to advance the measurement of library and network effectiveness/quality by the publication of several useful performance evaluation manuals.6 The Van House, Weil and McClure; McClure and Lopata; Poll; Sacks and Whildin; and Clarke books were sponsored or published by major professional organizations, designed for use in all academic libraries and widely field-tested. They provide detailed guidance about data collection for a variety of measures of effectiveness (and cost benefit/efficiency measures in the McClure and Lopata manual), however very few impact measures are included in these sources. The findings described in the multiple stakeholder research studies, such as Cullen and Calvert, and field-tested manuals referred to above, reveal what academic library constituent groups perceive to be important input and output performance indicators. As such, they are candidates for key "enabling" indicators, those which make it possible for the achievement or success of valued institutional outcomes, such as student development of information literacy skills and faculty research and publication. While I do not feel that it's necessarily desirable or possible to determine a standardized set of performance indicators, it is important to identify domains or areas of effectiveness/quality that all academic libraries include in assessment. Within these domains the use of a variety of field-tested and multiple constituencies-generated performance measures seem the most promising. A library's selection of pertinent performance measures, though, must be based on their connections to supporting valued institutional outcomes as articulated in local campus mission and goals statements and strategic plans. The following selected outcomes are considered important for most all institutions, as they were represented in three or more standards/criteria sources; the complete listing can be found in the forthcoming article. This section is organized by three categories of outcomes and enabling outputs: 1. Learning outcomes and enabling instructional outputs/good practice criteria. 2. Faculty/ academic staff research/scholarly productivity and professional development outcomes and enabling outputs. 3. Institutional viability and vitality outcomes and enabling outputs, which include outputs related to recruitment, retention, assessment and librarian contributions to academic affairs. Cross references in parentheses are made to selected performance indicators in the section which follows this one. I. Learning Outcomes 1. All graduates are information literate, prepared to be lifelong autonomous learners able to effectively identify, access and use a variety of resources; proficient with appropriate information technologies; and able to evaluate and apply information to meet academic, personal and job-related needs. (See performance indicators I.A. 1.- 8.) 2. All graduates have the knowledge and skills to conduct an effective job search. (See performance indicators I.C. 1. and 2.; IV.B. "number of hits various Web pages receive;" and V.B.) 3. All graduates have an understanding and appreciation of cultural diversity. (See performance indicators I.B.1., I.C.3. and III.3.) 4. Graduates pursuing post-baccalaureate study possess the knowledge and skills to succeed in graduate/professional programs. (See performance indicators I.A.4.- 5. and II.3.) Enabling Instructional Outputs and Good Practice Criteria 5. Undergraduate education, specifically the general education program, requires students to develop and use designated information competencies. (See performance indicators I.A.1. - 5.; and 7.) 6. Sufficient and appropriate library, network and other information and learning resources, equipment and services are provided and/or made accessible regardless of physical location or format, and integrated into educational programs by required usage in courses across the curriculum. (See performance indicators I.A.2. and A.7; IV. A. and B. selected measures and all measures in V.) 7. A holistic computing environment supporting direct-user access for all academic staff and students, regardless of location or time, is effectively operating and reflects service linkages among complementary units providing library, computing/network services and other information and learning resources. (See I.A.1.; I.B.1.; II.1.; and IV.A. and B. selected network access and use measures; and selected measures in V. A. - C.) 8. New knowledge programs/products to improve independent and course-related learning and access to digital and multimedia information have been acquired and/or created locally by collaborations between library and other academic units. (See I.A.1.; I.B.1.; III. 1.; and IV. A. and B. selected access and use measures.) 9. The academic environment is conducive to learning, growth and an awareness and appreciation of multicultural diversity. (See I.B.1.; I.C.3.; III.3.; V.B. and C.) 10. Effective instructional practices are employed, such as peer group interaction, problem-solving assignments, use of appropriate instructional technology, and other active learning methods that increase the quality of student involvement in learning. (See I.A.7., I.B.1. - 2.; and II.1.-3.) 11. Assessment plans, procedures and processes are in place to evaluate and improve the quality and effectiveness of learning, teaching, advising, and counseling. (See I.A.8. and III.5.) II. Faculty/Academic Staff Research/Scholarly Productivity and Professional Development Outcomes 1. Faculty/academic staff are active professionally and contribute to research, scholarly/creative works and community service. (See performance indicators II.3. and II.4.) 2. By means of various professional development activities, faculty and staff maintain excellence in job performance. (See performance indicators II.1. - 5.) Enabling Outputs/Criteria 3. Sufficient and appropriate library, network and other research and training resources and services are provided and/or made accessible regardless of physical location or format. (See performance indicators II.3.; IV. A. and B. selected access and use measures; all measures in V.A., B., and C.) 4. New knowledge programs/products to improve independent and course-related learning and access to digital and multimedia information for independent and classroom use have been acquired and/or created locally by collaborations among library and other academic units. (See I. A.1. and I. B.1.; III. 1. and IV. A. and B. selected access and use measures.) 5. Faculty and staff professional development programs are effectively operating. (See II.5.) III. Institutional Viability and Vitality Outcome 1. Student and faculty/staff recruitment and retention rates meet institutional targets and staffing needs. (See performance indicator III.3.) Enabling Outputs/Criteria 2. Campus revenue is sufficient to support educational programs and other academic support operations. (See performance indicators III.2. and V.A.) 3. Campus-wide assessment plan(s) and procedures, developed by appropriate segments of the institution and reflecting the needs of the institution's constituencies, are effectively put to use to advance institutional mission and goals. (See performance indicator III.5.) 4. The campus governance structure includes appropriate faculty, staff and students in its committee memberships and contributes significantly to campus operations and programs. (See III.1.) 5. Institutional units cooperate and the institution collaborates, as appropriate, with neighboring K-12 schools, community colleges and other organizations to improve education at all levels. (See IV.A. measure, "description of and results of cooperative resource-sharing agreements and contracts with external information and document providers, along with statistics and user perceptions...") Performance Indicators, Measures, and Other Supporting Documentation The next section provides both a conceptual framework to structure assessment efforts and a listing of specific performance indicators and measures from which to select for generating evidence of progress toward and/or accomplishment of valued institutional outcomes. This framework of five assessment domains -- 1. learning outcomes and enabling instructional outputs; 2. faculty/academic staff teaching effectiveness, scholarly productivity and professional development; 3. institutional viability and vitality; 4. access, availability and use of learning and teaching resources; and 5. infrastructure inputs -- reflects the valued institutional outcomes and outputs derived from the analysis of accreditation and ACRL standards' major themes. Separating the infrastructure types of indicators from indicators of access and use is helpful conceptually, but when measuring and reporting the results of these indicators, it may be more useful to connect data on access/availability and use to the descriptive information about collections, facilities and equipment wherever possible. Both quantitative and qualitative types of measures are included, as building a culture of evidence involves a cumulative, multi-methods approach. Sections IV. and V. below include representative indicators readily found in the performance evaluation manuals referenced above, and for that reason they are purposefully not as well developed as those described in sections I. through III., which were the project's primary objective to define and describe. Even though methods of data collection are not explicitly specified, the performance indicators suggest certain methods and the reader can find guidance in several of the previously referenced performance evaluation publications. For example, the phrase "perceptions of faculty and students" suggests the use of some type of survey, focus groups, or interviews to generate the data. In effect the selection of method is always driven by the performance indicator being measured. The listing below is not intended to be comprehensive; nor should readers assume that all of these measures are necessary for every institution. However, all five assessment domains should be represented by performance indicators selected to address local situations. Most noteworthy is the emphasis in many of these performance indicators on measuring and describing impacts/effects, results and contributions. Indeed, most of these indicators do not require data to "prove" that such and such factor caused such and such effect or outcome; rather the data obtained from these performance measures are correlational, and as such they indicate direct connections and document significant contributions of academic libraries and librarians. I. Learning Outcomes and Enabling Instructional Outputs A. Information Literacy 1. Description of, number of student participants and their perceptions of effectiveness and benefits of independent learning opportunities related to information literacy (e.g. locally produced tutorials/instructional software; reference transactions involving substantive teaching; term-paper or other individual research advising sessions; training videos; Web-based instruction; printed guides). 2. Documentation of the extent and effects of the integration of library and network resources use within academic programs and across-the-curriculum. For example, provide data on the number, type, and results of information literacy-related degree requirements, course requirements and assignments in each academic program. Results might include the number of students successfully completing assignments or courses, as well as actual student performance measures (see #3. which follows) and/or their perceptions about performance. 3. Grades, scores and student perceptions of their performance on assignments requiring evidence of specific information literacy competencies. The use of student self-evaluation, search logs or journals, course portfolios, and tests are suggested. 4. Longitudinal data for same sample of students comparing freshmen's or sophomore's rating of their level of confidence about being able to perform specific information literacy skills to that of seniors or recent graduates. 5. Perceptions of recent graduates about how their information literacy skills training/experience from undergraduate study contributes to their success in graduate/professional programs. 6. Success in applying information literacy skills on the job as perceived by alumni and employers. 7. Description of the information literacy program's reach and effects, including such measures as participation rate in formal and informal instruction/orientation and information literacy course completion rate and average grade per FTE student population. 8. Copy of the information literacy assessment plan, which includes: a. description of expected information literacy outcomes/competencies for general education and other academic programs and how competency or proficiency is determined; b. performance indicators for measuring student progress and achievement from college entrance/transfer-in to graduation; and c. demonstrated application of good assessment practices, such as faculty involvement in developing plan. B. Independent and Collaborative Peer Learning Opportunities 1. The number of, description of the relationship to institutional goals, and student/faculty perceptions of benefits of exhibits, programs (e.g.lecture or film series), and electronic or multi-media programs that are acquired or produced/co-produced by library/learning resource units. 2. Number of hours students spend studying in the library and doing library/network-based assignments. Data could be organized by academic program and correlated with average GPA. C. Co-Curricular Environment 1. Success in job seeking, as indicated by survey of recent graduates' perceptions of usefulness of library and network resources and library sponsored/co-sponsored workshops, exhibits and services. 2. Number, description of, and student perceptions about the benefits of library, computer and related information technology work experience programs and internships for students. 3. Description of the library sponsored or co-sponsored exhibits, programs, and other learning resources focusing on understanding and appreciating multicultural diversity. II. Faculty/Academic Staff Teaching Effectiveness, Scholarly Productivity & Professional Development 1. Student course evaluation ratings of the use and quality of active learning strategies, such as required use of library and network resources; group projects, problem-solving assignments, etc. 2. Student and faculty ratings of librarian teaching effectiveness, especially active learning strategies. 3. Perceptions and experiences of faculty/academic staff about the effects of network services/resources, such as: has the network changed the way you teach or do your job? If yes, how? Has the use of the network affected the quality of teaching material used in the classroom? Has the network affected the way you do literature searches, conduct research, communicate, or publish? If yes, how? 4. Description, use of, and faculty and student evaluations of the benefits of Web-based and other teaching support innovations that librarians have produced or co-produced with faculty. 5. Description of the academic staff development program which includes statistics about the number, type and effects of training and other learning opportunities for maintaining and improving the education and skills of librarians, disciplinary faculty and other academic staff involved with information and instructional technology. 6. Total number of grants secured, publications, presentations, creative works and community service projects, with the number and percentage of those which benefited from the use of library and/or network resources and services. III. Institutional Viability and Vitality 1. Quantitative and qualitative summary of the results of librarian membership on campus committees and their collaborations with disciplinary faculty and other academic staff, particularly describing products or outcomes relating to teaching, collection development, information technology planning and assessment, and other activities that contribute to institutional mission and goals. 2. Description of library's success in bringing gifts, donations and external funding to campus. 3. Number of minority staff/students recruited to work in the library/learning resource units. 4. Perceptions of all campus stakeholders about if and how the network affects institutional image. 5. Copy of the the library/learning resource units'assessment plan(s), as well as examples of questionnaire items that are included in other campus units' assessment instruments related to library and network resources and services. IV. Access, Availability and Use of Learning, Teaching and Research Resources A. Access/Availability Measures: proportion of collections/materials included in computer catalog; extent of and ease of access to library catalogs and databases; interlibrary loan/document delivery fill rate and turnaround time; number, description of, and results of cooperative resource-sharing agreements and contracts with external information and document providers, along with statistics and user perceptions of derived benefits; match of hours open and/or electronic resources available with user needs; speed and accuracy of reshelving of materials; and acquisition speed. B. Use Measures: (includes reference and other user assistance services that facilitate use): user satisfaction/success rate in finding and obtaining desired materials; number of remote and non-remote logins to opac and other networked resources per capita; number of searches made from remote and non-remote terminals per capita; counts of the number of hits library-maintained Web pages receive; total number and/or percentages of faculty/students/staff visiting library-produced parts of the CWIS; entrance gate counts per FTE and/or number of sign-ups for group study rooms; number of courses and students by academic program requiring use of library and network resources; circulation and in-house use of media data per FTE user, organized by academic program/major and category (freshmen, faculty, etc.) of borrowers; extent of use and satisfaction with reference assistance. V. Infrastructure -- Essential Inputs A. Human and Fiscal Resources: number of professional and support staff and how they are deployed to support campus and library mission and goals; ratio of reference and instruction/training services staff to known users and/or potential users; and expenditures connected to academic program benefits, documenting costs of unmet demand/need. Could include comparisons to peer institutions. B. Adequacy of Collections and Learning Resources (also see IV. A. above): statistics, organized wherever possible by academic program or broad discipline, of the number of locally held or accessible collections and learning resources, and a description of how these support the curricular, co-curricular and faculty research needs; currency of library materials; provision of multiple copies of high use items; flexibility of budget to respond to new subject areas; adequacy of library collections compared with peer institutions; and description and results of periodic collection evaluations. C. Facilities/Equipment: description of extent of network, its components, number of public access stations, number and percentage of classrooms, student labs, residence halls, and faculty offices that have access to campus network, and number of dial-in and other access modes for off-campus students and staff; user perceptions of the quietness of the study environment; number of seats and group study/work rooms per FTE and stakeholder perceptions about adequacy of such space; documentation on equipment (e.g. photocopiers, computer workstations) replacement and maintenance, along with user satisfaction ratings about adequacy. Conclusions In summary these points remain essential to the improvement of assessment: In collaboration with other key constituent groups, develop a library assessment plan that focuses on performance indicators that contribute to valued institutional outcomes. Look for other assessment instruments and opportunities on campus in which aspects of library/network performance can be included, such as entering freshmen, senior or alumni surveys. Establish benchmark data for key performance indicators, then document progress, change and achievement as part of the culture of evidence. Inventory and make use of relevant, available institutional and library data to complement other data collection methods so that a multi- methods approach is used. Organize and present data and findings in ways that are more meaningful to faculty and academic administrators, such as grouping data items by academic program or broad disciplines and describing how students benefit. REFERENCES CITED 1. For example, see Alexander Astin, "What Matters in College," Liberal Education 79 (Fall 1993): 4-15; Elizabeth A. Jones, National Assessment of College Student Learning: Identifying College Graduates' Essential Skills in Writing, Speech and Listening, and Critical Thinking: Final Project Report (Washington D.C.: Office of Educational Research and Improvement, U.S. Department of Education, 1995); A Preliminary Study of the Feasibility and Utility for National Policy of Instructional "Good Practice" Indicators in Undergraduate Education: Contractor Report (Washington, D.C.: Office of Educational Research and Improvement, U.S. Department of Education, 1994); and Rowena J. Cullen and Philip J. Calvert, "Stakeholder Perceptions of University Library Effectiveness," Journal of Academic Librarianship 21 (Nov. 1995): 438-48.Up 2. These roles and functions have been primarily drawn from Chris D.Ferguson and Charles A. Bunge, "The Shape of Services to Come: Values-Based Reference Service for the Largely Digital Library," College and Research Libraries 58 (May 1997): 252-265; and Carla J. Stoffle and Karen Williams,"The Instructional Program and Responsibilities of the Teaching Library," New Directions for Higher Education 90 (Summer 1995): 63-74.Up 3. See Paul B. Kantor, Objective Performance Measures for Academic and Research Libraries (Washington, D.C.: Association of Research Libraries, 1984); Charles R. McClure and Cynthia L. Lopata, Assessing the Academic Networked Enviroment: Strategies and Options (Washington, D.C.: Coalition for Networked Information, 1996); Frederick W. Lancaster, If You Want to Evaluate Your Library, 2nd rev. ed. (London: Library Association Publishing, 1993).Up 4. Charles McClure, "Measurement and Evaluation," in American Libraries Association Yearbook, vol. 9 (Chicago: ALA, 1985): 195.Up 5. McClure and Lopata, Assessing the Academic Networked Environment, 5.Up 6. For example, see Tobin de Leon Clarke, "Output Measures for Evaluating the Performance of Community College Learning Resources Programs: A California Case Study," in Advances in Librarianship, vol. 17, ed. Irene P. Godden (New York: Academic Press, 1993), 175-202; Charles R. McClure and Cynthia L. Lopata, Assessing the Academic Networked Enviroment: Strategies and Options (Washington, D.C.: Coalition for Networked Information, 1996); Roswitha Poll, ed., Measuring Quality: International Guidelines for Performance Measurement in Academic Libraries (Munchen: K.G. Saur, 1996); Patricia A. Sacks and Sara L. Whildin, Preparing for Accreditation: A Handbook for Academic Librarians (Chicago: American Library Association, 1993), 39-74; Nancy A. Van House, Beth T. Weil, and Charles R. McClure, Measuring Academic Library Performance: A Practical Approach (Chicago: American Library Association, 1990).Up

ArchivesResearch Award ArchiveAwards