HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 

THE VALUE OF EXTERNAL EVALUATION TO SCHOOL'S 
SELF-ASSESSMENT 

Heikki K. Lyytinen 

Provincial Government of Western Finland 
 
 

WHAT IS EXTERNAL EVALUATION? 

External evaluation can be defined in this context as evaluation of school by an expert, or expert body, from outside the school community. Scriven (1991) defines an external evaluator as a person who is not involved with the project or programme to be evaluated, or when the target is a whole work community, is not a member of that community. An external evaluator can be an evaluation specialist (e.g. a consultant), an expert of the field in question (an educationalist), or an external body representing local, regional or central school administration, for instance. 

External evaluation may be invited by the school and targeted according to its needs, or it may be performed ex officio, as a part of school authorities' responsibilities. External and internal evaluation may also be merged into a reflective dialogue without clear distinction between external and internal, to increase the reliability and validity of the results. In such case evaluation becomes more democratic and unhierarchical. 

To make external evaluation effective, the school should have a genuine need for it. If evaluation is felt as interfering with the school's autonomy or as control-seeking administrative activity, no fruitful dialogue will be achieved. From the effectiveness point of view, it is important that the external evaluation and the internal self-assessment of the school can be combined, so that the school community sees external evaluation as a source of beneficial support (see Laukkanen 1996). Especially when introducing the system of steering by results, the role of school administration in conducting external evaluations is emphasized, as evaluation is focused on educational outcomes and on the cost-effectiveness of educational investments. 

In steering by results, school authorities and their policies are to support schools in their efforts for increased effectiveness. This entai ls surveying the need for support, as well as evaluating the adequacy and appropriateness of the measures taken, i.e. estimating the efficiency of the means chosen for reaching the goals set. This means, of course, that school administrators need to have sound knowledge about how to enhance educational outcomes as well as adequate evaluation skills in general. 
Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 

ON THE RELATIONSHIP BETWEEN SELF-ASSESSMENT AND EXTERNAL EVALUATION 

In self-assessment a community as well as its individual members are inevitably fettered by their own perspectives, criteria and whole culture of operations and practices. Consequently, the information produced by such an evaluation is always one-sided, in one way or another, as a result from subjective bias. A tendency towards a positive identity, i.e. seeing own actions solely in a positive light, may distort the evaluation data. On the other hand. evaluation based on over-critical or negative attitudes, in turn, brings along uncertainty, which prevents dealing with differing views which would be essential for open evaluation. Then the information produced by evaluation fails to bring about a constructive dialogue (Lyytinen 1995 ; Nikkanen 1992). 

Self-assessment can thus be attached with certain reservations. Therefore it would be good to complement self-assessment with external evaluation from time to time. Though it must be noted that this approach, also, often draws mainly on self-assessment data of the school community. Self-assessment and external evaluation should be seen as complementary approaches which can be joined together to form a seamless whole. The validity of external evaluation is essentially linked with that of self-assessment. In order to use these appproaches as complementary to each other, their respective strengths and weaknesses should be matched together for optimal balance (Nikkanen & Lyytinen 1996 ). 

The central strengths of self-assessment can be listed as follows: 

  • Effectiveness, as it involves personal and independent decisions, and because it is about activating, grass-root level practice. The information yielded is fairly easy to put into good use. 
  • Continuity, as it makes up a natural part of operations planning and the regular ourse of events. Very useful for formative purposes. 
  • Learning effect, as it is essentially about an intellectual exercise on experience, and learning from it. 
  • Characteristics of personal responsibility, which reinforces commitment. 
  • Flexibility and precision, as it can be tailored and targeted according to the specific needs and prior knowledge of the community. 
  • Information production, which serves various needs of school development and management. 
    •  Correspondingly, the weak points of self-assessment are as follows: 
  • It is fettered by the system of social relations within the work community, so that taking a due distance from the evaluation targets may be more difficult. 
  • Subjectivity and narrow viewpoints particularly in contexts where no comparative information is available. 
  • Emphasis on the positive characteristics and identity of the work community, so that critical views become more difficult to take. People prefer to see things in a positive light, because negativeness increases anxiety within the work community. 
  • Inadequacies in the evaluation thinking and capacity of the work community. Evaluation skills can only be learnt through practice. The Finnish educational culture has not always sufficiently prepared people for this, however. 
    •
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 
    From the overall effectiveness point of view, external evaluation should remove or mitigate the weaknesses of self-assessment. External evaluation, after all, has better possibilities to examine the situation free from the social restrictions of the work community, although, strictly speaking. it may not be fully objective, either. Evaluators' dependence on their customers and prevailing «evaluation markets» may decrease the degree of objectivity. Negative assessments may be avoided in order to keep up relations with the customer, by trading off some ethical considerations related to evaluation. 

    An external evaluation can be commissioned by the target organization itself, or, for instance, by the maintainer of the school. Best results are obtained when the target organizations are being involved on a voluntary basis. 

    In contrast with the weaknesses of self-assessment, external evaluation can provide the following benefits: 

  • Increased objectivity in evaluation. An external evaluator, though, may have a ndency to approach the target from the angle of general findings elsewhere, even when the target itself would not provide sufficient evidence to this effect (cf. Scriven 1991). 
  • Comparable reference information, when the information from a case evaluation is examined with relation to and mirrored against a wider context. 
  • It helps to be at a distance from the functions and processes, which the target organization has created and is closely involved in, and which are affected by the social relations within the work community. In self-assessment social relations may have a subtle impact on the focus so that some areas are left in shadow; external evaluation may cast some light on these issues, as well. 
  • It partly offsets the above-mentioned fallacy of positive identity characteristic to a work community. 
  • Serves pre-emptively when identifying and solving problems looming on the horizon. 
    •  In order to complement self-assessment and enhance the self-directiveness of the work community in the best possible way, also the general drawbacks of external evaluation must be taken into account. Traditional administration-oriented external evaluation, especially, has been characterized e.g. by the following qualities: 
  • Being detached, bureaucratic and disintegrated. 
  • Evaluation process takes place at the level of external evaluator only, without real self-assessment by the work community. 
  • Changes are mainly cosmetic by nature; no deeper changes in the operation culture or in the commitment and responsibility within the work community, no creative solutions or innovations. 
    •  These drawbacks can be partly avoided by adopting a process-centred approach in external evaluation. It is founded on an adequately many-faceted self-assessment by the work community with due methodologcal guidance. 

    For schools, in particular, external evaluation is justifiable as lending support to self-assessment. The phenomena schools deal with on a daily basis are highly complex. The fundamental issue, learning, 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 
    consists of a multitude of explanatory factors. Also the mechanisms through which these factors affect learning are manifold. Similarly, the aims of schooling are elusive or indefinite, even. There is often uncertainty in schools about which methods would yield the best results. But also the concept of result is ambiguous here. When dealing with such problems, dialogic external evaluation may provide useful supplementary information to self-assessment (Lyytinen 1993, 124-125). It can offer valid, research-based criteria which a school itself has no time nor possibilities to sort out. 

    Evaluation taking place at the school system level is essentially external by nature. We can therefore speak of external educational evaluation at national level. It can bear significance to self-assessments by individual schools, as well, although it primarily seeks to develop the system-level fundamentals of schooling, including the educational steering system. External evaluation at this level provides broad situational information, which may serve as a basis when defining the school's ends and means, the where-to's and how-to's. The purpose of a national evaluation system is specifically to produce and disseminate many-sided, updated and reliable information on the education system and its conditions, functioning, outcomes and effects in terms of its main bjectives, taking also the international context into account. 

    In the context of steering by results, in particular, external evaluation is justifiable on the following grounds: 

  • Schools' aims are elusive and partly indefinite. Therefore dialogue with an outside expert may prove helpful in outlining the objectives. 
  • There may be uncertainty in the school community as for which would be the methods, processes and lines of action to yield the desired results. A dialogue with an expert can give support and reduce uncertainty in strategy formulation. 
    •  Also the results of the school are hard to define and partly ambiguous. External evaluation and discussion may help recognize the problems related to educational outcomes and clarify the essence of results in this field. 

    EXTERNAL EVALUATION AND DEVELOPING A SCHOOL 

    Self-assessment should be naturally integrated in a school-based development process. As noted above, self-assessment can be regarded as a development-strategic approach in itself. Thus, self-assessment should support the school's development cycle in its various phases. 

    What, then, could an external evaluator's contribution be to this process? In the following, I will focus on the capacity of external evaluation to enhance the development process of a school. 

    1. When examining the work roles and work-related emotions, especially, an external evaluator may help take distance from the events. Also when exploring professional difficulties, certain distance is often necessary. 
    At the initial stages of the development a picture as detailed as possible of the current situation and developmental needs is required. Such information can be gathered naturally by means of self-assessment. At this stage also the tasks, and their respective appropriateness, of the members of the work community are addressed. External evaluation can clarify the questions and answers brought up, and 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 
    help choose evaluation models and approaches. External evaluators, for their part, can help understand the nature of developmental needs and possibly place them in wider contexts. 
    2. It is usually difficult to set priorities solely on the basis of an self-assessment. If people have not learnt to deal with matters together so that various views are taken into account, it may prove difficult to find development targets and objectives people are willing to work for. By means of appropriately formulated questions and through interpreting openly the situational information available, an external evaluator can help people see what is more and what less important. 
     
    3. The capability of perceiving and taking new perspectives is restricted by the a particular point of view. Moreover, these viewpoints are adopted also by the opinion leaders. Especially when the atmosphere at the workplace is reserved or even suspicious, people dare not propose any novel viewpoints. New ideas are more readily accepted if they come from an external evaluator. This requires, of course, certain confidence between the evaluator and the work community in question. 
    4. In sketching the outlines of the school's future and setting the development objectives an external evaluator may help finding realistic alternatives. A visionary approach to planning, in particular, is susceptible to unrealistic estimations, so that the limitations related to resources or time needed to achieve the goals may not receive due consideration. Of course, the director of the school in his or her own role is bound to pay attention to these matters, acting in a sense as an external evaluator. But the director, too, is similarly involved in the process, which may make it difficult to see the wood for the trees. 
    5. Development ideas and objectives, as such, do not mean anything would change in practice. Self-assessment within a school and the ideas brought up this way do have an impact on attitudes and values, though. Such a mental exercise may, indeed, have great importance for the practical implementation, as it can advance commitment to the development scheme and lead to its further elaboration. 
    Developmental implementation is a process, where the development ideas, programmes, or schemes are put into practice (Lyytinen & Uusitalo 1991, 64 ; see also Nikkanen & Valta 1985 ). When proceeding to the implementation stage, an external evaluator together with the school community can consider e.g. potentially impeding or facilitating factors. This way the chances for successful implementation can be improved. Some developmental impediments are listed in Appendix 1 (cf. Lyytinen and Uusitalo 1991, 64 ). If the situation is the reverse of these descriptions, it is favourable to the implementation. An external evaluator may help especially in identifying the critical points, and in selecting between various alternative solutions, by providing evaluative feedback both on the implementation plan and on the set-up itself. 
    6. The degree to which development objectives have been met can usually be determined by self-assessment, if the objectives have been clearly defined and if people are used to 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 
    the practice. However, external evaluators can help obtain the «evidence» of the results achieved. They can also help develop evaluation methods for these purposes. 
     
    7. Whereas developmental implementation means moving from theory to practice, getting new experiences as well as learning to do things differently; establishing new practices means their permanent adoption. Establishing an activity that was set as an development objective can thus be regarded as a process where new and lasting values, attitudes, practices, and structures are «created». This process is often difficult to evaluate without external guidance. It takes at least an introductory, or a run-in stage which increases people's evaluative capacity, as they learn to understand the essence of establishing a new practice. 

    ON THE ETHICAL VALUE BASE OF EXTERNAL EVALUATION 

    Benefits from external evaluation can by no means be taken for granted, especially if it fails to meets the ethical requirements for evaluation. To meet these requirements external evaluation must be performed with high expertise and applied with responsibility. 

    Evaluation and development activities may even prove counter-effective as the school involved may regress and come to a deadlock so that the work community is not considered as «good» as it used to be. If such an undesirable result seems permanent, the nature of the evaluation and development activities may have been ethically questionable, especially when nothing is done to remedy the situation. An error may have occurred when interpreting the evaluation information. Things may have been brought up in a way the school community has found unbearable. Perhaps the evaluator has lacked adequate expertise and skill, after all, to evaluate the inner state of the school community. 

    The developmental proposals subsequent to the evaluation involve the members of the community and their behaviour. When we are dealing with human behaviour and related control, change or influence in general, numerous ethical considerations become pertinent. The same is true for influencing the development of a school as a whole, as well. 

    The ethical considerations faced in developmental work can be traced back to the question whether these evaluation and development activities are right and valid in terms of their aims, methods, and outcomes. By the same token, we can ask what are the motives underlying these activities, and what is their justification and value basis? 

    Ethical considerations, in this context, are confined to the developmental evaluation of schools. Evaluation is always a process of interaction, which is usually started and maintained in order to cause some changes in the school's functioning. The interaction takes place between the evaluator and the `client' organization. Evaluation seeks to enhance the development of the school so that it could better fulfill its basic tasks. Evaluation produces process-like, long-term repercussions. These potentially positive or negative consequences should be estimated and anticipated in evaluation. If evaluators are unaware of their influence and the values embedded, their actions lack adequate justification. 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 
    The central ethical rules concern respect for the autonomy of the school community, professional secrecy, recognition of the limitations of one's own expertise, and keeping the evaluation task purely professional. One must be able to justify the evaluation on the grounds of professional ethics. Evaluation is unjustifiable unless it is of high standard in ethical terms. Other important value considerations in the evaluation of school communities are as follows (cf. Lyytinen & Parviainen 1987 ): 
    1. Evaluation is based on respect for the prevailing organization culture of the school community. 
    2. Evaluation is founded on confidential collaboration, voluntariness and equality. 
    3. Evaluation is customer-oriented. The goal is optimal development of schools and institutes from their own starting points so that it also serves broader social and educational policies. 
    4. Evaluation means giving expert support for making the right choices, plans and decisions in the various stages of development. 
    5. Evaluation calls for knowledge - as good as possible - of the organization and its culture, and their investigation by various methods available (questionnaire, interview, observation etc.). 
    6. Evaluation and its consequences must be subjected to a continuous, critical review. 
    Besides theoretical and methodological knowledge on school communities' evaluation and development, an external evaluator thus needs to have high professional ethics, insight on social and educational policies, and enough independence and freedom of action in the various stages of an external evaluation. 

    ON THE RESPONSIBILITY OF AN EXTERNAL EVALUATOR 

    The general viewpoints mentioned above are also closely related to the ethical responsibility of an external evaluator. A central responsibility here concerns the evaluator's own expertise. Evaluation must be confined to matters that lie in the scope of the evaluator's expertise. Then the evaluation relationship will not turn into a generalized dependence relation. The evaluator need not be, nor try to be, omniscient, although social pressures or temptations to this effect are often present in an evaluation situation. Evaluators should avoid raising result expectations which are too great relative to their expertise. If evaluators create ill-founded «over-expectations» or consciously let them emerge, their behaviour is unethical. 

    The issue of evaluators' expertise and professionalism becomes even more complex, if the evaluation is associated with evident exertion of power. The evaluator may be competent in certain specific area of expertise, but prove unable to adapt his or her intervention appropriately and with due expertise in all respects. An evaluator who relies greatly on a pre-programmed intervention may, for instance, unintentionally define the problem areas and development targets of an organization too narrowly, according to his or her own limitations of time and competence. (Cf. Walton & Warwick 
    1973     ). 

    Evaluators' responsibility extends over to customers' expectations also when these expectations are not due to the evaluators themselves. The customers may expect certain kinds of evaluation findings 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 
    significant to the school's image, for example. Such findings do not necessarily advance the development of that school, however. One can easily end up in a «social pitfall» unless one feels due ethical responsibility for the potential consequences and acts accordingly. 

    Evaluators are also responsible for not letting their own strivings, motives and needs to have an undue impact on the evaluation. If the evaluation turns into a playground for the evaluator's own needs, the activity has then no ethical justification. If the evaluator's own inner reality gets mixed up with the reality of the organization to be evaluated, it will prevent critical monitoring and reviewing of the subsequent process of development. 

    Appropriate external evaluation entails, however, sufficient dedication to the customer. Evaluation that purports to develop a school organization is a time-consuming process. The evaluator is responsible for ensuring that the evaluation relationship will not break unnecessarily. If the evaluator lacks either the will or opportunity to be involved in the development to the extent desired, the matter should be settled as clearly as possible, already at the initial stages of the evaluation relationship. 

    Another important ethical problem is the psychological dependence on the external evaluation which easily develops in the customer. Such dependence can sustain even if the original aims of evaluation were already reached. This must be eliminated so that the evaluator and the customer make an explicit contract, agreeing upon the point where this customer relation ends. The contract can also hold preliminary terms on meetings, e.g. with respect to their frequency and nature. 

    Evaluators, for their part, are also responsible for establishing customer relations with confidence. Confidence derives from genuineness and from honest striving for the development of the organization in question. Confidence also builds on the fairness of evaluation, where information acquisition, various choices and decision-making, as well as supportive measures, take place in an unbiased fashion. 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 

    References 

    Laukkanen, R. 1996. Oppilaitos hyötyy ulkoisesta evaluaatiosta. Aikuiskasvatus 3, 211-218. 

    Lyytinen, H.K. & Parviainen, H. 1987. Konsultoinnin eettisistä näkökohdista. In K. Hämäläinen (eds.) Konsultoiva työtapa ja koulun kehittäminen. Oulun yliopiston kasvatustieteiden tiedekunnan opetusmonisteita ja selosteita 23, 57-65. 

    Lyytinen, H.K. & Uusitalo, R. 1991. Uudistuva yläaste yhdessä kehittämällä ajan tasalle. Helsinki: Valtion painatuskeskus. 

    Lyytinen, H.K. 1993. Sisäinen ja ulkoinen arviointi koulun kehittämisessä. In K. Hämäläinen, R. Laukkanen & A. Mikkola (eds.) Koulun tuloksellisuuden arviointi. Helsinki: Painatuskeskus, 108-128. 

    Lyytinen, H.K. 1995. Johdatus oppilaitoskohtaiseen itsearviointiin. In B. Kilpinen, K. Salmio, L. Vainio & A. Vanne (eds.) Itsearvioinnin teoriaa ja käytäntöä. Helsinki: Cosmoprint, 37-55. 

    Nikkanen, P. 1992. Kohti oppivaa ammattioppilaitosta. Hämeenlinna. Ammatillinen opettajakorkeakoulu. Tutkimuksia 9. 

    Nikkanen, P. & Lyytinen, H.K. 1996. Oppiva koulu ja itsearviointi. Jyväskylän yliopisto. Koulutuksen tutkimuslaitos. 

    Nikkanen, P. & Valta, M. 1985. Ammattioppilaitoksen sisäinen kehittäminen. Ammattikoulujen Hämeenlinnan Opettajaopiston julkaisuja 20. 

    Walton, R. E. & Warwick, D. P. 1983. The ethics of organization development. Journal of Applied Behavioral Science 9, 681-698. 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment. 
    HEIKKI K. LYYTINEN: THE VALUE OF EXTERNAL EVALUATION . . . 

    Appendix 1 

    Impediments to development efforts 
    1. The development project means unreasonable extra work load in comparison to the previous practice, so that people lose their interest in development. 

    2. The development scheme is so novel, strange or difficult to carry out in practice that it involves much of prejudice and too many new things to be learnt. 

    3. Too great a turnover of the staff, especially among the key persons, during the project. 

    4. The development project does not get the internal, external, psychological, technical, or possibly financial support it would need. 

    5. The development project proves, in one way or another, irrelevant or meaningless from the standpoint of the school's basic function. 

    6. Failure in the management of the development process, where e.g. the resistance to change, selling the development ideas, and the relationship between the new and the old systems have not been taken into account sufficiently. This usually means that there are too few people giving back-up to the project, and the organization's overall commitment to the ideas has remained scanty. Inadequate knowledge about the nature and critical stages of development may decrease the potential for situational management and thus result in a superficial approach to developmental management. 
    Hannu Jokinen & John Rushton (Eds.) 1998. Changing Contexts of School Development - The Challenges to Evaluation and Assessment.