EVALUATION AND TESTING IN ESP

Presented by:
Sitti Fatimah Saleng, Fifin Naili Riskiyah, & Herwindho Mukti

State University of Malang

EVALUATIONS

Definitons

Pauline robinson in his books defines evaluation as the discovery of the value for some purposes. Other clearer definitions is proposed by alderson and murphy who writes that evaluation is concerned with describing what is there and placing some value on what is found.

There are some scholars stated that the evaluation has been neglected in ESP. Swan suggest eight possible reason for the underdevelopment of evaluation in ESP. Some of the possible reasons according to swan are the shortness or even one-off nature of ESP courses, the time consuming nature of evaluation and the lack of any felt of evaluation.

A basic distinction can be made between formative evaluation and summative evaluation. Formative evaluation is carried out during the life of a course or project and the results obtained can be used to modify what is being done. Summative evaluation is carried out when the course or project is finished. Therefore, the result of the summative evaluation will be used as the basis to take decision whether to repeat the course or not, or whether the money has been well spent or not.

A further distinction can be made between process and product evaluation. A process evaluation may be concerned with teaching and learning strategies or processes, and admnistrative and decision making process. A product evaluation will look, in particular, at students product such as examination results, essays, etc. However, lynch writes that the strongest approach to evaluation is the one that combines as many methods, qualitative and quantitative, as are appropriate to the particular evaluation context.

Why carry out an evaluation

Since both formative and summative evaluation can be undertaken to provide data as input to possible changes, thus evaluation can be used, in particular, as part of quality control. In other cases an evaluation may function as a source of information and experience. A further reason for carrying out an evaluation may be to ensure that money is being or has been well spent. 

What is the subject of the evaluation exercise

An evaluation exercise can be wide ranging or very limited in scale. The subject may be a whole ESP projector course, or just one or some aspects. Long suggests that the formative evaluation typically look at such factors as teachers’ and students’ attitudes toward a curricular innovation, or at the usability of new instructional materials. A summative evaluation may look at virtually all aspects of a programme, possibly with a particular interest in cost-effectiveness.  

Who carries out the evaluation

As with needs analysis, a basic choice must be made between the outsider and the insider. For large aid-funded projects an outsider is typically brought in. Outsiders may be consultants, inspectors or administrators. Weir and Roberts (1994) say that for many years ELT project evaluations were carried out by external ELT specialists. They are considered experts because they have spent years teaching, running projects or training, with the drawback of not having being involved with the programme to be evaluated. Outsiders can bring a different perspective and encourage participants to discuss.

The insiders, in the other side are most likely to be the course designers and the teachers, but students also may be involved. Insiders may be teachers, students, staff and anyone else closely involved in the development and implementation of the programme. Weir & Roberts (1994:23) observe that “insiders have far greater experience of the situation and are aware of the history behind development”.

How is the evaluation carried out?

 The procedures for data collection in the evaluation are similar to that for need analysis. In conducting evaluation, several considerations should be noticed such as cost effectiveness in terms of time and money and appropriateness for the situation. Data collection and data analysis often take longer time than we imagined at first. Therefore, everything should be well-managed.

·         Questionnaires

Obvious problems with questionnaires are their length, possible misinterpretation of questions by respondents and failure of respondents to return them. Questionnaires are given to students, ex-students, ESP teachers and subject specialists.

·         Checklist

It is shorter than questionnaire and is used for small-scale aspects of a program, such as speaking skill.

·         Interviews

It is very time consuming but provides the opportunity for more extended exploration of the points. To make the interview effective and efficient, it must be well-planned.

·         Observation

Observation is done during the classroom process. Before doing observation, we must prepare appropriate observation sheets. Besides observing the present students, it is also possible to observe the past students to see the effectiveness of the course.

·         Discussions

Class discussion can form a regular part of the program, the topic, the content or the general approach to methodology, etc.

·         Records

Records of the project or program such as the original planning documents, statements of needs, aims and objectives are ideally kept from the beginning. These reports are used for evaluation purposes i.e. in the form of checklist which can be filled right after the lesson is over or in the form of diaries of students’ and teachers’ experiences on a course.

·         Assessment

It includes tests and examinations, the evaluations of students’ projects, written work, etc which are carried out by teachers or evaluators. Brindley suggested that students are better to be involved in the assessment process by having student-self and peer-assessment (as also Lewcowicz and Moon stated).

Evaluation is a crucial and integral part of the instructional process that teacher may be more or less permanently conducting evaluation in the methods or elements of the designed program. An important consideration is reliability of the instruments or techniques and the validity of the results. To increase the degree of reliability and validity, evaluators should do some cross-checking of the results by triangulation. It is targeting the same point by means of two or more techniques of evaluation, i.e. having evaluation, class discussion and a rating scale to find information on classroom methodology.

TESTING

ESP testing is a relatively neglected area. In 1983, Alderson and Waters wrote that ‘only limited numbers of articles have been published in the area of ESP testing, but nowadays, there have been a lot of journals related to ESP testing. ESP tests are related in content, themes and topics to particular disciplines, and involve a higher degree of language specificity.

ESP tests are more concerned to present learners with tasks that involve them in reading, listening to, speaking or writing the target language, and evaluating how well they can do this. Of course, an important component in assessing how well somebody can use English is how accurately they can produce or understand texts written or spoken in the language, but the key to this assessment is to present learners with tasks that resemble in some way the sort of things they may have to do with the language in real life. Therefore, the ESP approach in testing is based on the analysis of learners’ target language use situations and specialist knowledge of using English for real communication.

Performance-Based Testing

Performance-Based Testing is an examination approach wherein candidates must interact with real or simulated systems. PBT puts certification candidates in situations where they must use their knowledge, demonstrate their skills, solve common problems, and perform troubleshooting tasks

Depending on characteristics, Language and other tests can be:

• Norm Referenced Test:  provides information about an individual’s relative rank with reference to other individuals who have taken the test” (Bachman in Robinson, 1991: 73)
• Criterion Referenced Test:  test score are reported and interpreted as reference to specific context domain or criterion of performance.

Tests for EGP are typically norm-referenced, while ESP are typically Criterion-referenced test. One of the tasks of the ESP test designer is to determine the exact nature of the criterion for judging adequate performance. Theoretically, an ESP test would consist of performance in a real life situation. Allison and Webber in Robinson give a good review of performance-based tests for communication skills courses for EAP where the primary aim of the activity is to reinforce teaching and learning.

How specific should the content be?

When devising an ESP test one should aim at creating good and dependable measures of language ability which need to:

• the content must be valid,
• the topic form must be paralel, like IELTS test (have three paralel forms (life sciences, physical sciences and arts).
• be as authentic as possible,
• provide accurate and reliable measures of language ability,
•  have beneficial effects,
• be practical and economical in terms of administration, time, money and Personnel.

One of the prevailing principles of ESP testing is that tests should contain tasks that mirror faithfully those of the candidates' target language use situation

Predictive validity

The IELTS and TEEP test examine the linguistic proficiency which related to academic success. The linguistic performance can be the basis for a prediction about future performance. The better language proficiency of students, the better their academic success.

Non-EAP tests

• The London Chamber of Commerce and Industry's tests. It applies proficiency test at the beginning of course and achievement test at the end of the course.
• The business English tests. It applies placement test to know level of students and achievement test to know the result of teaching-learning process.
• Temporary Registration Assessment Board (TRAB) / Professional Linguistic Assessment Board (PLAB) of the UK General Medical Council Test. It has sub test which measure linguistic proficiency, medical test, and interview test (simulated doctor-patient interaction), etc.

The link between testing and teaching

• To give evaluation and feedback to the teaching and learning process
• To help teachers, syllabus constructors, material’s developers
• Spoken test can encourage students to think about their purpose in learnig English and their attitudes toward English.
• Tests can give learners a sense of accomplishment and a feeling that the teacher’s evaluation matches what skills and knowledge have been covered.
• A good test will automatically have beneficial effects on classroom instruction. Other variables such as teacher competence, motivation and innovation, the climate of the school, socioeconomic status of pupils and teachers combine to exert an equally important influence on what goes on in an ESP classroom.

Conclusion

ESP tests should be based on direct evaluation of language ability in acts of communication (reading, speaking, writing, and listening) and assessment of content knowledge in situations and activities, which are as engaging and realistic as possible for learners. The primary goal of ESP tests is to obtain information about the learner's specific purpose language ability. In present day, it is increasingly important not only to be able to use a foreign language, but also to be able to demonstrate that one can use it at the level required by employers, schools, or universities. Considering this, ESP examinations are gaining more and more attention, and are becoming an indispensable tool in the modern educational society.

References:

Robinson, Pauline. 1991. ESP Today: A Practitioner’s Guide. UK: Prentice Hall

Tratnik, Alenka. 2008. Key Issues in Testing English for Specific Purposes. SDUTSJ