The SEALLL project approaches its completion. This doesn't mean that no questions are left to be answered, nor that we now know all there is to be known about self- evaluation in adult life long learning. It does mean however that the time has come to look at its outcomes and products.

SEALLL produced a manual that can be downloaded for free from the project website from October 1 on, in English, German, Turkish, Dutch, Polish, Lithuanian and Swedish. This manual offers some theoretical background and practical material to support adult learning organisations in their self-evaluation processes.

The SEALLL philosophy

In SEALLL self-evaluation is considered a matter of learning. Self-evaluation is the process of discovery learning or of experiential learning of professionals during their work. Individually and collectively they are searching for ways to identify the quality of their work, what might better be sustained and what should be improved or changed. Self-evaluation should be organized as learning processes. If one takes that seriously it implies that self evaluations should be:

• Motivating people
• Provide them with new information and insights
• Offer them feedback on their actions
• Provide opportunities to share and analyse their experiences
• Offer opportunities to experiment
• Give feedback on the results

Self-evaluation feeds individual, collective and organizational learning processes. These processes in mutual interaction lead to quality improvement. This improved quality shows in two ways. People will learn how to improve the work they have done thus far and secondly they may be challenged to change their work if self-evaluation reveals a need for that.

Self-evaluations implicitly or explicitly are dialogues
In the SEALLL project an additional element of self-evaluation has been identified, elaborated and included in the philosophy. It is the element of dialogue. No matter if actual dialogues are included in self-evaluation, one may consider self-evaluation a dialogue. It is always either an inner dialogue, or a dialogue with learners, colleagues, funding agencies, stakeholders. The nature of such dialogues can vary a lot. Some dialogues are shared attempts to understand the mechanisms of adult education better, some are meant to account for ones actions, others are to be considered negotiations or disputes.

Instruments and methods

The SEALLL manual also presents a lot of instruments. The general idea behind these instruments and the recommended methods is that traditional (self) evaluation instruments tend to be made in such a way that respondents individually answer questions asked by others. Furthermore traditional instruments tend to be providing little room for creative productive answers but rather ask for specific information. For the learning process SEALLL promotes quite the opposite: we want self- evaluation instruments also to be productive and collective. None of the instruments mentioned below are to be avoided as such, but for purposes of professional learning it is a challenge to use at least a few instruments from the right side and the right side lower quadrant as included in the scheme below .

You will find more examples in the SEALLL manual

Practical guide for facilitators Who initiates or facilitates self-evaluation? In the SEALLL project we discussed whether it would be realistic to expect self- evaluation just to emerge form practice. We felt it would not. There may often be a need for someone who launches the idea and then helps the self-evaluators elaborate it. That is why we decided to develop a set of practical guidelines for facilitators of self-evaluation. It shows for each step of the process of self-evaluation how a person may have an input, or provide support without frustrating the idea of self-evaluation. This practical guide for facilitators of self-evaluation may be found as an appendix of the manual. The big challenge in making this practical guide was to see to it that it would not turn the actual self-evaluators into marionettes, but into professionals who would feel empowered to cut their strings and develop their autonomy.

Introducing self-evaluation and learning from it involves much more than ticking boxes on questionnaires, prepared by others. Therefore the SEALLL team tries to offer an approach and tools for organisations to introduce and set up their own self-evaluation, create their own instruments and take this further in an organisational learning process.

Sealll tries to guide the 'initiators' in the process of conducting their own evaluation, this can be learners, teachers, developers or management. Starting from this point SEALLL has created a format presenting all the steps in the process of introducing and executing self-evaluation and follow up activities. The SEALLL site offers a series of examples of self-evaluation processes in a wide range of adult education settings created by member adult education organisations. These examples are not meant to be exhaustive and only serve as a source for inspiration and suggestions for 'initiators' in similar conditions or with similar aims.

Please keep in mind: this material is to be adapted to your own conditions, you cannot solve everything via one instrument, set your own priorities and keep it simple.

SEALLL products are available on the project website!

• Landcommanderij Alden Biesen, Bilzen, BE, coordinator:
  Guy Tilkin: guy.tilkin@alden-biesen.be
• PLATO, University of Leiden, NL:
  Jaap Van Lakerveld: laker@fsw.leidenuniv.nl
  Selma van der Haar: haarsvd@plato.leidenuniv.nl
• Soros International House, Vilnius, LT:
  Daiva Malinauskiene: daiva@sih.lt
• Kwasimodo, Brussels, BE:
  Anita Caals: anita.caals@kwasimodo.be
• Centre for Flexible Learning, Söderhamn SE:
  Tove Elvelid: tove.elvelid@soderhamn.se
• ISIS Institut für Soziale Infrastruktur, Frankfurt am Main, DE:
  Karl Mingot: mingot@isis-sozialforschung.de
• The Bauer-Messner EvaluierungsKEG, Graz, AT:
  Christa Bauer: christa.bauer@aon.at
  Angelica Petrovic: Angelika.Petrovic@inode.at
• Fundacja Centrum Edukacji Obywatelskiej, Warsaw, PL:
  Kasia Batko: kasiab@ceo.org.pl
  Iwona Kania: iwona.kania@ceo.org.pl
• Akdeniz University Center for Continuous Education, Antalya, TR:
  Gunsiray Košun: gunsiray@akdeniz.edu.tr
  Burhan Ozkan: bozkan@akdeniz.edu.tr

We would like to invite you to the SEALLL final conference on self-evaluation in formal and non-formal life long learning settings. This conference takes place on the 21 September in the Conscience building, department of Education in Brussels and is organised in cooperation with EPOS, the Flemish LLP programme agency.

SEALLL aims to improve the quality of teaching and learning and the quality of organisation and management in LLL by promoting and supporting self-evaluation. The project wants to help all 'players' in LLL-organisations self-evaluate their teaching, learning and management.

9 partners from 8 countries - in cooperation with 30 'member institutions' - participated in SEALLL. You can find more information on: www.sealll.eu

This conference targets all actors in formal and non-formal adult education ( management, curriculum developers, inspection, teachers, trainers ... ) involved or interested in quality care and self-evaluation.

09.30: reception & coffee

• 10.00: Welcome: SEALLL and quality care in adult learning: Jos Van Rillaer, Administrator-General, Agency for Arts and Heritage
• 10.15: Welcome on behalf of EPOS, the Flemish LLP agency: Annemie Dewael, EPOS general director
• 10.25: SEALLL, the project: Guy Tilkin, project manager Alden Biesen, SEALLL coordinator
• 10.45: The SEALLL approach to self-evaluation: Jaap Van Lakerveld, PLATO, Leiden University (NL)
• 11.15: Self-evaluation as a dialogue for learning, Christa Bauer, Qualitas KEG, Graz (AU)
• 12.00: Presentation of the project website: Tove Elvelid, Centrum för flexibelt lärande, Söderhamns kommun (SE)

12.30: Lunch

• 13.45: Practical examples of self-evaluation processes developed by SEALLL member institutions in small workshops (Nederlands and English).
• 14.45: Panel discussion: Self-evaluation and learning experiences.
• 15.45: Reception and distribution of the manual.

Address: Consciencegebouw, Albert II laan 15, Brussel.
5 min. walk from North station.
Fee: 12 Euro (documents, lunch, manual and reception included).

Subscription: by mail to: myriam.swinnen@cjsm.vlaanderen.be
please give: your name, name & type of your organisation,
mail address and by banktransfer (name!) of 12 Euro on Dexia nr. 091 2224016 29

SEALLL products are available on the project website! Go to: www.sealll.eu

SEALLL is a Grundtvig 1 project, part of the European SOCRATES educational programme and aims to improve the quality of teaching and learning and the quality of organisation and management in LLL by promoting and supporting self-evaluation. The project wants to help all 'players' in LLL-organisations self-evaluate their teaching, learning and management.

In this newsletter we give some examples of self-evaluation following the processes advocated by Sealll.

Self-evaluation can have a wide angle of items: it is not only about the content of training and methods. All elements of the organization can be self-evaluated as long as the initiator is the owner of the process. Self-evaluation can be done at all levels of the organization: management level, organizational level, operational level or even personal level.

Sometimes people think that evaluation is not possible with certain participants: that we cannot bother them with our questions and that they are not willing to participate. If we are more creative we can find a way to make self-evaluation more attractive and find out that people are pleased to give their comments as long as we can give them the feeling or the proof that something will be done with it. This means that self-evaluation does not start with the application of an instrument but long before, on the moment that you consider doing a self-evaluation or even when you determine a lack of information. Find out below how an Austrian organization dealt with it: Self-evaluating a Counselling Situation

Sometimes we want to collect too much information at the same time and we forget about the participants who have to fill in the often very long questionnaires. Before starting to think about the information we want to gather, maybe we better define criteria which we think that are important for the participant. Kwasimodo, a Belgian organization, evaluated the way they evaluated their training sessions and transferred their 4-pages questionnaire into a more sexy self-evaluation instrument. Find out how in example 2.

“I am not against it, not at all. But can it be done? The women who come here are in a desperate situation. Can they be bothered to fill in a questionnaire about the quality of my work when they do not know how to feed their children or pay their rent? I'd feel terrible about it.” – The counsellor looks at me doubtfully and with a look at her senior colleague: “I do understand that I should be able to account for my work and of course, there's always room for improvement and feedback would be very helpful in some aspects.”

This dialogue took place at the very beginning of a counseling session in a department of one of our member institutions in the SEALLL-project. It is an institution for giving support to women in crisis situations. The basic principle of the institution is to offer a very gentle service to the women. For that reason it seemed difficult for the team to develop a way of self- evaluation which did not ask too much of the clients. So finding a way of combining self-evaluation with the basic principles of the institution was the topic of the counseling session. And this is how it went:

"I suggest looking at the aspects of her work she would like to have feedback on. We agree to follow the planning instrument in our SE-grid. So, what are possible goals for the evaluation, what kind of information would help her achieve these goals? Who does she need this information from? - Most importantly, she suggests, we should look at the environment of the counselling situation. It is taking place in a women's café to encourage women who are shy in asking for help. “But of course, this also means it isn't private, people from other tables listen in, sometimes there are women with children and they are making noise, the radio can be loud - it is very difficult and unsatisfying”. - I look at her. “You want to have evidence for something you already know” and at her boss, “How much evidence or how strong evidence would you need in order to help improve the situation?” – “There is no money, we cannot reconstruct the building, but if the evidence is strong enough we will find a solution”. While we are trying to work out how many complaints of women would be needed to really make a change, the counsellor cannot be stopped from sharing a few of her experiences. In the following dialogue the counsellor and her boss start developing a solution which both get very excited about.

We start looking at some other aspects. It turns out the counsellor would like to know about the quality of the counselling but shows me with disgust the feedback questionnaire her predecessor used. “I cannot imagine asking a woman in need all these questions. Besides, she is not an expert, how can I expect her answers to be on this level? What I would like to know is whether the woman feels better about her situation after the counselling.” I suggest using symbols as an instrument. I have got some objects in my bag and put them on the table. “Can you choose one symbol on how you feel on the solved problem of the counselling environment?” Both women are happy to choose and describe the reasons why they did so. I ask them how they feel about the instrument. They feel good, because, they argue that it made them look at their own feelings. The counsellor says she would really like to work with this instrument. Together with her boss we agree that on the minutes she has to write following a counselling situation, there should be a feedback item saying: Symbol chosen at the end of the meeting/ reason why…. It would be feedback for the counselling while at the same time allowing the client to look at her own feelings.

The counsellor now comes forward with another thing she would be really interested in. “I am recommending certain institutions to my clients, but I do not get feedback on whether or not these institutions are the right ones for the problem and whether or not they are supportive and gender sensitive enough. It would be really helpful to know more about this aspect. Mostly, I never hear from the women again and it can mean they are doing well but it can also mean the opposite. To know more about this could help me improve my work.” So we are trying to work out an instrument that does not mean too much trouble for the clients but will give the counsellor the necessary information. We end up with a postage prepaid postcard with closed questions, maybe some smilies, and room for additional remarks. The counsellor says she would be happy to draft the actual questions herself and send them to me for feedback."

I think this is a really good example showing that you can adapt (self-evaluation to almost every work situation) without offending the principles or main idea of your work. It only takes creative ideas. I have recently heard from our member institution, saying that they have already started working with the instruments, and that they are really satisfied because their clients respond well to them.

Kwasimodo is a Belgian non-profit making organisation, supported and subsidised by the Flemish Community, Department of Culture. Kwasimodo aims to promote and implement quality management within organizations in the social and cultural sectors. Self-evaluation is an important aspect within quality management. Kwasimodo has successfully developed a full set of tools to evaluate the degree of quality management existing within an organization.

1. Why did we plan to develop new tools?

Continuous improvement is the central theme in TQM. Evaluation is therefore an essential process in identifying and planning improvements. As an expert organization representing TQM, Kwasimodo has always been actively involved in evaluating the training and consultancy sessions delivered to client organisations.

However, a critical analysis identified that Kwasimodo had not used recent evaluation data to develop and implement improvements within the organisation. There were several reasons for this:

• the extended evaluations were not completed correctly or extensively by clients
• the answers only gave a partial view on the evaluation items
• many of the evaluation questions were not relevant or wholly concerned with the delivery of Kwasimodo services

2. What were the aims of the evaluation?

The trainers at Kwasimodo wanted to gain information concerning the effectiveness and the efficiency of the training sessions delivered. In addition, the trainers also wanted to generate evaluation data to demonstrate to Kwasimodo board members that they were continuously improving their training sessions.

It was important to the trainers at Kwasimodo that they implemented good practice and that they had a feedback mechanism which clients could use to offer objective feedback and evaluation data. Clients participating in the training sessions were also keen to offer objective information relating to the quality and delivery of training in the knowledge that this information would be used to enhance services and future training sessions.

Within this context it is understood that the nature of the evaluation questions should be concerned with the delivery of services (in this case training) and not with other issues that neither the trainers nor Kwasimodo has no control over. The evaluation should provide clear feedback in relation to the efficiency and effectiveness of the training sessions. It should also respond to the indicators and priorities of the client organizations represented at the training sessions and to the expectations of the participants.

3. How did we negotiate the criteria?

Step One: Definition of the criteria for evaluation

This was achieved by listing the different types of training sessions and by defining appropriate indicators using a system of process management. This process resulted in 5 items to be evaluated:

• the content
• the learning process
• the general satisfaction
• the intention to continue to work with Kwasimodo
• the level of active participation of the clients

Step Two: Who will be questioned and what are their expectations?

In order to maintain a quick and simple evaluation system which would not be regarded as arduous or time-consuming for clients to complete, each evaluation item was developed using only three evaluation questions. Furthermore, the evaluation items may be combined, according to the client group providing the evaluation feedback.

Step Three: Developing the evaluation items

Kwasimodo used the five evaluation items to develop various evaluation models, according to the client target group. The evaluation item relating to content was regarded as fundamental to all evaluation models whereas the remaining four evaluation items related to process and delivery and could be combined, depending on the nature of the client target group.

4. Tips and tricks

• Just consider what you really need to know and what you will use effectively to improve the training schemes.
• Be honest in reading the results of the evaluation. It is useless to change or interpret the results.
• Give feed back to the client participants about the evaluation results and how these will impact future training sessions.
• Evaluation tools are not permanent. Change them whenever necessary
• Try to keep the evaluation process interesting and innovative for both the client participants and the trainers. Be creative.
• Evaluate continuously the evaluating method so you can be sure that the method is still relevant and efficient.

SEALLL products are available on the project website!

• Landcommanderij Alden Biesen, Bilzen, BE, coordinator:
  Guy Tilkin: guy.tilkin@alden-biesen.be
• PLATO, University of Leiden, NL:
  Jaap Van Lakerveld: laker@fsw.leidenuniv.nl
  Selma van der Haar: haarsvd@plato.leidenuniv.nl
• Soros International House, Vilnius, LT:
  Daiva Malinauskiene: daiva@sih.lt
• Kwasimodo, Brussels, BE:
  Anita Caals: anita.caals@kwasimodo.be
• Centre for Flexible Learning, Söderhamn SE:
  Tove Elvelid: tove.elvelid@soderhamn.se
• ISIS Institut für Soziale Infrastruktur, Frankfurt am Main, DE:
  Karl Mingot: mingot@isis-sozialforschung.de
• The Bauer-Messner EvaluierungsKEG, Graz, AT:
  Christa Bauer: christa.bauer@aon.at
  Angelica Petrovic: Angelika.Petrovic@inode.at
• Fundacja Centrum Edukacji Obywatelskiej, Warsaw, PL:
  Kasia Batko: kasiab@ceo.org.pl
  Iwona Kania: iwona.kania@ceo.org.pl
• Akdeniz University Center for Continuous Education, Antalya, TR:
  Gunsiray Košun: gunsiray@akdeniz.edu.tr

Life Long Learning situations are often less formal and less structured than school and higher education. Therefore quality care, self-evaluation, self-regulated learning and setting up the institution as a learning organisation is even more important in LLL than in formal learning situations.

SEALLL is a Grundtvig 1 project, part of the European SOCRATES educational programme and aims to improve the quality of teaching and learning and the quality of organisation and management in LLL by promoting and supporting self-evaluation. The project wants to help all 'players' in LLL-organisations self-evaluate their teaching, learning and management.

SEALLL is NOT another suite of ready made evaluation instruments with indicators to check and boxes to tick.
SEALLL focuses on self-evaluation as a learning process and the partnership is convinced that a culture of self-evaluation and quality care and ownership of processes and outcomes are vital for an evaluation process to have a chance to lead to change. Therefore a bottom up approach and guiding the 'initiators' in the process of conducting their own evaluation is the lead motive of SEALLL.

The SEALLL partnership sees self-evaluation as an evaluation process that is self-initiated, internally organised and self-regulated. It should aim at the professionalisation of decision-making, and improving the realisation of the organisation's own objectives and the quality of its education.

The tool envisaged for the project will start from a modular framework where 'self-evaluation as a dialogue in a multiplayer situation' will be the key-concept. A dialogue between staff, teachers and learners within the institution and a dialogue between the institution and relevant external actors will be the starting point for SE.

• To help all 'players' in LLL-institutions self-evaluate their teaching, learning and the management of their organisation
• To help LLL-institutions set up their institution as a learning organisation at all levels.
• To promote a bottom-up approach to self-evaluation in LLL.
• To ensure a broad European approach to the creation of this material.
• To focus on the use of self-evaluation data, the analysis, the consequences.

• A modular frame work of dimensions, indicators and criteria for self-evaluation of quality in LLL settings that enables potential users to design their own evaluation process
• A website with the modular frame, appropriate tools, guidelines, self-evaluation instruments, on line SE tools, examples of good practice in this field etc.
• Training sessions and in-service days at regional/national level.
• A pool of experts, trainers and speakers in this field. (for organisers of national or international training events)
• A manual for teachers/trainers and staff with guidelines, methods and techniques to introduce and implement self-evaluation in adult learning.
• An international conference for dissemination and validation of the material.

• Teachers, trainers, heads and administrative staff of formal, non-formal and informal adult education institutions
• Teacher trainers
• Educational and socio-cultural departments of local/regional/national authorities (policy makers).
• Programme developers
• Adult learners

Via the project website you can find:

• Guidelines for introducing self-evaluation in your organisation
• Some theory on the SEALLL approach to self-evaluation
• A format for starting processes of self-evaluation and creating your own instruments
• Examples of self-evaluation processes in a wide range of LLL settings, ordered according to initiator and subject.

SEALLL also offers info and training sessions in the partner countries and speakers and workshop leaders for conferences set up by other organisations.

SEALLL products are available on the project website!