Abstract

One of the principal challenges facing metropolitan universities is how to explain and account for their contributions to their communities. The metropolitan dimension to these institutions' missions is not well understood by the public, nor is it well documented by universities themselves. This article explains how and why California State University, Sacramento and Portland State University have taken very different approaches to confronting this challenge and offers suggestions and conclusions based on these experiences.

Assessing the Metropolitan University Mission

Nancy B. Shulock

Kathi A. Ketcheson

Everyone engaged in higher education administration is well aware of the pressures to be accountable. Accrediting agencies, policy makers, and others are asking universities to account for the value of the educational enterprise. The principal tool for addressing accountability is assessment, which is carried out at two levels. Assessment of student learning outcomes has received the most attention. At least as important is institutional assessment--an accounting of the degree to which a university fulfills its broad institutional mission. For metropolitan universities, where service to the region is an integral part of the mission, institutional assessment is vital but especially challenging. This article explores how two metropolitan universities have undertaken this challenge in very different ways.

Assessing the metropolitan mission is challenging for two reasons. First, the metropolitan university mission is not well understood (a fact which occasioned the founding of the Coalition of Metropolitan Universities). Policy makers routinely look to traditional indicators of quality that are inappropriate to our institutions, such as four-year graduation rates, entering SAT scores, and federal research grants. These measures fail to capture the essential role that metropolitan universities seek to play in their communities, such as regional economic development, contributions to lifelong learning, and first generation access to college. Even our own faculty members, who come principally from research universities with very different missions, may not fully appreciate the metropolitan dimensions of our mission. As a result, they may not orient their own teaching, research, and service to best match the mission. As those of us involved in institutional assessment attempt to promote understanding of the metropolitan mission, we encounter the second challenge: defining, operationalizing, and measuring the metropolitan mission.

California State University, Sacramento (CSUS) and Portland State University (PSU) have taken different paths on the journey to understand, communicate, and assess the metropolitan mission. These paths are reflective of a striking difference in institutional culture that belies the similarities of mission, history, geography, and institutional size (see Figure 1).

Figure 1

CSUS is a process-oriented culture, possibly a remnant of administrative-faculty distrust from the tumultuous 1960s and 1970s, as well as from its status as one campus in a very large system with a history of regulation. It is most comfortable when carrying out plans that have been developed and adopted by an agreed-upon process. PSU is an action-oriented culture, for reasons possibly related to the "trailblazer" orientation of Oregonians combined with PSU’s origins as an extension center for returning World War II veterans. It has tended to eschew formal planning processes in favor of strategic actions, which have catalyzed organization members around key ideas.

Culture is one of the metaphors that Gareth Morgan (Images of Organization, 1997), advises public managers to use in order to "read" their organizations as a basis for facilitating organizational change. CSUS and PSU have succeeded in adopting and assessing a metropolitan mission in large part because they have adapted their approaches to their cultures. CSUS has relied extensively on its Strategic Plan and integrated planning process to shape the understanding and the assessment of the mission. PSU has chosen to proceed without a formal strategic plan and to organize its work around its graduate and professional programs, a strong liberal arts curriculum, and reforms in general education that bring the University into direct and continuing contact with the community. Both approaches are described below.

The "integrated planning process" at CSUS brings together planning, assessment, and resource allocation. Institutional assessment and the University budget are driven by the eight themes of the Strategic Plan. The Planning Council, with representation from all campus constituencies, meets bi-weekly throughout the Fall semester to review assessment reports that address priorities established for each Strategic Plan theme. The Council also reviews key performance indicators (KPIs) that have been approved for each theme. Subgroups of the Council are empowered to analyze assessment data (the reports and the KPIs) and recommend to the full Council on implications of the data for next year's budget priorities. By the end of the term, the Council recommends "resource priorities" to the President which, once approved, become the basis for budget requests. In the Spring, the Council reviews the budget as presented by the three Vice Presidents, who must explain how it addresses the "resource priorities."

Through this process we have established a "culture of evidence" whereby planning and budget decisions derive from assessment data. We have accomplished this by grounding all our institutional assessment in the themes of the Strategic Plan. With the Plan providing the purpose for our assessment and the questions we need to answer, we have avoided the common problems of getting lost in the data and providing data for data's sake. For example, we administer an extensive set of standardized surveys to students, faculty, staff, and alumni. But we never report survey results out of context of the Strategic Plan. Instead, we merge into each assessment report those survey data and relevant institutional data that deal directly with the issue at hand. In this way, we can always answer the "so what?" question.

The specific tools of the integrated planning process are described in Figure 2.

Figure 2

Our Strategic Plan communicates the metropolitan mission of CSUS throughout its eight themes, which are Teaching and Learning, Academic Programs, Scholarship, Pluralism, Enrollment Planning, Campus Life, Public Life of a Capital University, and Institutional Effectiveness. We build in overlap in communicating the metropolitan mission because the mission is still not widely understood insofar as how it affects the curriculum, pedagogy, scholarship, and the life of the University on and beyond the campus. The themes, the associated planning priorities, and the KPIs are all selected with the metropolitan mission firmly in mind. Some examples of KPIs with strong metropolitan university significance are:

• Responsiveness of teaching to individual needs
• Availability of a variety of instructional methods
• Regional emphasis of academic program
• The diversity of the student body relative to the region
• Faculty address issues of pluralism through the curriculum

Figure 3 illustrates the whole process for the Enrollment Planning theme.

Figure 3

An important part of the metropolitan mission is the impact the university has on its region. We capture this dimension most directly in our "public life of a capital university" theme, whose goal is: "to establish partnerships and programs of mutual benefit to the University and the Sacramento region in the areas of human and social services, cultural life, economic development, and public policy issues of regional and statewide significance." Assessing this theme has been difficult because none of the standardized surveys or institutional data are relevant. To assess how we are viewed in the region, we need to go to the region. We are in the early stages of designing an instrument for this kind of assessment. Before we got to this state, however, we went down an unfruitful road that is worth describing. Our first theme assessment paper was an inventory of our efforts in the region. This approach had three problems. First, we hadn't defined "public life" sufficiently to inventory it. Second, an inventory is out of date the day after you finish it. Third, and most importantly, an inventory is not an assessment of outcomes--it is just a listing of inputs or commitments. The "so what?" question was untouched.

We have since developed a framework that defines the theme as the ways in which our instruction, scholarship, service, and other programs contribute to the human/social services, cultural, economic, and public policy dimensions of the region. The framework further differentiates between our commitment to doing certain things (i.e. inputs) and the public benefit of having done them. While we may choose to inventory select areas if we suspect problems, our assessment effort will focus only on the public benefit, or outcomes, part of the framework. To a large degree, this will be dependent on designing an instrument that asks key stakeholders about the impact of the University. This will be a valuable extension of our previous assessment efforts.

While PSU has been in the forefront of national attention regarding its reform of general education, it has not been a leader in assessment. A culture of change, initiated in the early 1990’s, moved the institution forward in a number of areas. Change efforts capitalized on the entrepreneurial spirit that has characterized the institution since its earliest days, encouraging the development of new approaches both in academics and administration. Efforts to develop processes for the assessment of teaching and learning, as well as institutional assessment, were slower to follow these innovations. By the close of the decade, however, these processes had begun to take shape.

A statewide property tax limitation measure, passed by voters in 1990, resulted in substantial budget reductions for Oregon’s colleges and universities. PSU eliminated academic units, programs, and faculty positions in 1991-92. At the same time, the Oregon University System imposed enrollment caps on each of the eight system institutions.

To manage uncertainty, the President initiated a strategic planning process for PSU, which included broad participation across the campus and its external community. The intent was to state a clear mission for the University and create a unique identity within Oregon’s university system that would help PSU focus its programs and activities toward specific goals. The Strategic Planning Committee identified four general themes: Metropolitan Context; Academic, Research, and Service Programs; Serving the Community and Today’s Students; and Managing the University’s Future.

Several factors constrained implementation of the plan. The document proved to be too ambitious in its goals and too specific in its language. It did not specify how assessment or evaluation would be carried out or which units or individuals would be responsible for the implementation of new activities. At that time, a climate for assessment did not exist on campus and the entrepreneurial nature of the faculty culture served as a counterweight to formal processes. Although some PSU faculty may not want to be thought of as innovators, but simply wish to do things well, it is clear that creative actions have been vital to PSU’s success. These actions have grown out of circumstance and culture and have not fit easily with our attempts at formal planning.

One important result of the Strategic Plan, however, was that the University adopted a mission statement that reflected PSU’s urban context and set a direction for the future. A plan for development of a University District to define the campus physically within the downtown area soon followed. At the same time, innovations in the undergraduate curriculum were emerging across the campus, outside of the formal planning process. Reform of the general education curriculum capitalized on PSU’s urban mission and context to institutionalize service learning and community connections. It coincided with reforms in science education and the innovative use of technology in large classes.

As PSU moves forward into the 21^st Century, the administration has articulated an "umbrella strategy." This strategy allows the administration to set broad goals and communicate clear messages about the mission of the institution, while leaving faculty, staff, and students the freedom to create and implement activities in support of the metropolitan mission. An overarching goal is to develop PSU as a model for a new urban university for the 21^st Century, capitalizing on our circumstances and environment, rather than attempting to emulate other institutions.

This will be accomplished in part through strategic efforts to tie academic and administrative initiatives to the budget. The Resource Allocation Model adopted by the Oregon University System in 1999 allows institutions more autonomy over their budgets. In 1999-2000, PSU initiated a strategic budget process involving direct participation by the Faculty Senate Budget Committee. The process includes a set of criteria linked to the broad strategic goals of the institution by which all budget proposals will be evaluated.

The University Assessment Council has begun work with departments and programs to develop assessment plans for the collection and analysis of teaching and learning outcomes. Assessment is intended to encourage teaching and learning excellence and is viewed by faculty and administration as a scholarly activity. Faculty attitudes toward assessment have changed dramatically since the mid-90’s, moving from substantial resistance to scholarly curiosity and general acceptance. Typical of PSU, the leadership of key faculty members has been crucial in bringing about this change, rather than any direct administrative action, beyond appointment of the Assessment Council.

Assessment results are fed back into teaching, curricular, and administrative activities where faculty and students can see direct results. Although attempts at program assessment have largely failed, classroom-based assessments and mechanisms for gathering feedback from students about their learning experiences have provided evidence of program effectiveness. Some of the techniques currently in use are classroom assessment techniques, focus groups, portfolio assessment, and student self-assessments. On the institution level, the Office of Institutional Research and Planning uses data from the student information system to track student participation, progress, retention, and graduation, and routinely links survey results with these data to gain insight on student satisfaction and characteristics. As the Resource Allocation Model is implemented, program assessment results will not be tied to resource allocation, but will be tied at the division level to the institution’s broad strategic goals.

In its struggle to thrive within the Oregon University System, PSU has relied on the creative thinking and strategic actions of its faculty and administration. While some faculty do not view their efforts as "innovative," it is clear that PSU is the institution it is today because of its entrepreneurial spirit and willingness to capitalize on its unique place in the Portland metropolitan region. PSU frequently has had to operate on short-term goals as resource, space, and political limitations have restricted its ability to develop long-term plans. Successful planning has been possible, even in the absence of formal processes, with the adoption of an umbrella strategy tied to a clear statement of the urban mission.

Our challenge as we move forward will be to maintain our innovative character while adopting processes that make sense in our culture. While strategic actions in support of our urban mission will continue to drive our planning decisions, we will be open to alternative models. Program review, university-wide assessment planning, and the systematic collection and analysis of data in support of the Resource Allocation Model will provide us with the information we need to assess our progress toward institutional goals.

The experiences at CSUS and PSU lead us to offer conclusions in three areas: communicating the metropolitan mission, understanding institutional culture, and using data to accomplish institutional assessment.

1. Mission Issues

We cannot overemphasize the importance of communicating the metropolitan mission in conversations both on-campus and off-campus. The lure and familiarity of traditional models of higher education are powerful. We should not assume that our constituents understand fully how the mission of the metropolitan university affects the academic program, pedagogy, scholarship, service, campus life, and community engagement. Nor should we assume that they understand why traditional means of describing and measuring institutional accomplishments are inappropriate. The traditional university has been accepted for hundreds of years; the metropolitan university model is brand new by comparison. Even our own faculty need constant reminding that where they work is not like where they were educated.

For both of the culture types represented in our two cases, a laser focus on the metropolitan mission is invaluable. At CSUS the mission is a constant focus of the documents and discussions used in the planning process. It is built into each theme of the Strategic Plan, reflected in the KPIs and associated standards, and discussed at biweekly meetings of the Council for University Planning. The metropolitan mission supplies the criteria for decisions regarding budget, academic program emphasis, new faculty hiring, and the University’s extracurricular offerings. At PSU the metropolitan mission is played out daily in the lives of every undergraduate student and those faculty who teach in the University Studies general education program. Having made the decision to orient the Senior Capstone course around community-based themes, the annual selection of new topics and identification of faculty and community partners requires consideration of how best to merge education with community objectives. Many graduate and professional programs, too, incorporate community-based research, internships, or practica. For both institutions, the focus on mission promotes understanding of the mission itself and of the need to engage in assessment for purposes of accountability.

2. Culture Issues

Both the process-oriented culture of CSUS and the action-oriented culture of PSU are variants of the "garbage can" decision processes that are widely believed to characterize decision making in universities and other complex organizations (March and Olsen, Ambiguity and Choice in Organizations, 1976; Rediscovering Institutions, 1989) In neither case does decision making follow ideal, "rational" models where goals are well defined in advance and appropriate means selected based on careful analysis of options and projection of consequences.

Even at CSUS, where decision making is closer to the "deliberate" end of Mintzberg’s deliberate-emergent continuum (Crafting Strategy, 1987), it is far from the classic, rational model. Most of the important planning questions cannot be answered with certainty even after careful analysis. What are the benefits of alternative teaching strategies to our various student populations? What kind of class schedule would best meet student needs? How can the curriculum best promote pluralism? What kinds of extra-curricular activities will increase students' bonds with the University and promote retention? CSUS made huge strides in its assessment efforts when members of the Council for University Planning stopped asking for certainty in the data or for clear-cut answers to the means-ends question and agreed instead simply to commit to work on selected issues where the data raised serious questions.

In this era of accountability and performance budgeting there is an unfortunate tendency to approach assessment as a rational science. Campuses strive to uncover the causal relationships between dollar inputs and programmatic outcomes and attempt to delineate goals, objectives, and strategies, and outcomes as if they can be rigorously controlled and manipulated. The experience at CSUS has shown that it is possible to honor the culture’s dependence on process while recognizing the fundamental messiness of assessment. The integrated planning process has been accepted as a collaborative, evolutionary effort to make the university a better place, based on a "culture of evidence." Textbook approaches have been steadfastly avoided as inconsistent with both the specific campus culture and the generic character of universities.

Similarly, PSU has learned through attempts at formal strategic planning that overly rational approaches are too specific and too ambitious for the task of assessment. The campus was able to let go of the ambitious processes and retain the core concepts. From there, the concepts served to organize the innovative activities that were emerging across campus. One by one, the campus implemented strategic actions, not strategic plans, that provided the focus for assessment. Rather than be driven by a process, assessment at PSU is built into each of the actions. Assessment has become an accepted part of the process of learning from doing.

Our experiences suggest that regardless of the specific culture, the success of institutional assessment rests upon two foundations: a commitment to use data to inform planning, and a demonstrated link between planning and budget allocations. Once campus constituents see that the flow of dollars is influenced by assessment, they will see assessment as a legitimate undertaking.

3. Data Issues

For those who feel at times like they are drowning in data, the most important lesson to draw from our experiences is the need to be very selective about using data. We should use only those data that have a clear purpose and answer questions that we want to ask. The mission and the corresponding planning themes supply these purposes and questions. It is easy to get caught up in data for data’s sake. In an open environment where data are shared, data beget data. Every question we attempt to answer with data can raise new questions that require more data. Before we procure the additional data, and before we provide the data in the first place, we need to ask the "so what?" question. We need to force those engaged in assessment to choose the best indicator or review the most meaningful measures. We can, as CSUS has done, use a small number of key performance indicators to judge progress at the "macro" level and probe additional measures only if we see any "red flags" in the overview measures. Clearly we need to collect vast amounts of data. But we don’t need to report more than a fraction of it in a regular manner.

A second data issue of paramount importance to metropolitan universities concerns the development of measures that can capture the public engagement aspect of our mission. We cannot find these measures in standardized surveys of students, faculty, and staff. Likewise, we cannot find them in our vaults of institutional data on ethnicity, enrollment, faculty workload, and budgets. We must develop new measures. We must go out into the community, ask how we are doing, and document it. Before we can do this, we must define what we mean by "public engagement," "community outreach," or whatever we choose to call it. This is no easy task but it must be addressed if we are to garner any legitimacy for the claims we make about the contributions of our universities to the greater community.

Finally, our experiences underscore the changes that have occurred in the role of institutional research (IR). As the focus has shifted from inputs and activity accounting to accountability for outcomes, IR staff members have become responsible for interpreting data in light of intended outcomes. This puts a premium on their collaboration with other constituents and other offices across the university whose interpretations are needed. In addition, the demands of organizing vast amounts of data to support assessment have led IR staff to innovate using information technology. Technology is essential both to organize the data and to communicate it. Assessment is not principally about collecting the data and building the culture of evidence. It is, most fundamentally, about opening lines of communication about the evidence so that the broader community can understand what their universities contribute and so that those contributions can continue to grow.

Suggested readings:

Mintzberg, Henry, "Crafting Strategy," Harvard Business Review, July-August 1987

Morgan, Gareth Images of Organization, 1999

Authors' Biographies

Nancy B. Shulock is Associate Vice President for Academic Affairs at California State University, Sacramento and a faculty member in the Graduate Program in Public Policy and Administration.

Kathi A. Ketcheson is Director of Institutional Research and Planning at Portland State University.