eLearning Innovations: Curriculum Testing

Now we will explore the most important types of decisions (tests) that must be made in most language programs: proficiency, placement, diagnostic, and achievement.

Making Decisions with Tests

The four different types of tests, proficiency, placement, diagnostic, and achievement are probably emphasized because they fit neatly with four of the fundamental types of decisions that must be made in language programs.

PROFICIENCY DECISIONS

Teachers sometimes find themselves in the position of having to determine how much of a given language their students have learned and retained.

General proficiency is being used to describe what the student should have attained by the time they finish the program. It is a decision that must be made by the administrators, teachers, and contract negotiators involved.

For example, TOEFL is an overall English language proficiency test that is widely used to judge students for admissions decisions. The proficiency levels of students when they enter the program must also be measured.

To contractual issues, entry and exit level proficiencies are crucial for understanding the overall boundaries of a program. What level of overall proficiency do the students have when they come to us? And what level will they have when they leave us? Answering these two fundamental questions will help planners in making many different types of curriculum decisions.

Checking at the beginning of the curriculum development process to see if program objectives are set at the appropriate level for the students is far more productive than wafting until after the program is firmly in place, at which point costly materials, equipment, and staff decisions have already been made.

However, such decisions must be made carefully because proficiency tests are not designed to measure specific types of language teaching and learning, and most definitely not the specific types of language teaching and learning that are taking place in a particular language center.

In short, proficiency decisions involve tests that are general in nature (and not specific to any particular program) because proficiency decisions require general estimates of students' proficiency levels. Such decisions may be necessary in determining exit and entrance standards for a curriculum, in adjusting the level of goals and objectives to the true abilities of the students, or in making comparisons across programs. Despite the fact that proficiency decisions are general in nature, they are nevertheless very important in most language programs.

PLACEMENT DECISIONS

Also relatively general in purpose, placement decisions are necessary because of the desirability of grouping students of similar ability levels together in the same classes within a program. Some teachers feel that they can do better teaching when they can focus in each class on the problems and learning points appropriate to students at a particular level.

Placement tests are designed to facilitate the grouping of students according to their general level of ability. The purpose of a placement test is to show which students in a program have more of, or less of a particular ability, knowledge, or skill.

The placement of students into levels may be based on something entirely different from what is taught in the levels of the program.

In short, placement decisions should be based on instruments that are cither designed with a specific program in mind or, at least, seriously examined for their appropriateness to a specific program. The tests upon which placement decisions are based should either be specifically designed for a given program (and/or track within a program) or, at least, carefully examined and selected to reflect the goals and ability levels in the program. Thus a placement test will tend to apply only to a specific program and will be narrower in purpose than a proficiency test.

ACHIEVEMENT DECISIONS

Students' achievement is the amount that has been learned. To make any decisions related to student achievement and how to improve it, planners must have some idea of the amount of language that each person is learning in a given period of time (with very specific reference to a particular program).

To help with such decisions, tests can be designed that are directly linked to the program goals and objectives. These achievement tests will typically be administered at the end of a course or program to determine how effectively students have mastered the desired objectives.

The information gained in this type of testing can also be put to good use in reexamining the needs analysis, in selecting or creating materials and teaching strategies, and in evaluating program effectiveness. Thus the development of systematic achievement tests is crucial to the evolution of a systematic curriculum.

In short, achievement decisions are central to any language curriculum. We are in the business of fostering achievement in the form of language learning. In fact, this book promotes the idea that the purpose of curriculum is to maximize the possibilities for students to achieve a high decree of language learning. The tests used to monitor such achievement must be very specific to the goals and objectives of a given program and must be flexible in the sense that they can readily be made to change in response to what is learned from them about the other elements of the curriculum. In other words, well-considered achievement decisions are based on tests from which a great deal can be learned about the program. These tests should, in turn, be flexible and responsive in the sense that their results can be used to affect changes and to continually assess those changes against the program realities.

DIAGNOSTIC DECISIONS

The last category of decisions is concerned with diagnosing problems that students may have during the learning process, This type of decision is clearly related to achievement decisions, but here the concern is, with obtaining detailed information about individual students' areas of strength and weakness.

The purpose is to help students and their teachers to focus their efforts where they are most needed and where they will be most effective. In this context, "areas of strength and weakness" will refer to examining the degree to which the specific instructional objectives of the program are part of what students know about the language or can do with it. While achievement decisions are usually centered on the degree to which these objectives have been met at the end of a program or course, diagnostic decisions are normally made along the way as the students arc learning the language. As a result, diagnostic tests are typically administered at the beginning or in the middle of a course.

In short, diagnostic decisions are focused on the strengths and weaknesses of each individual vis-à-vis the instructional objectives for purposes of correcting deficiencies "before it is too late." Hence, diagnostic decisions are aimed at fostering achievement by promoting strengths and eliminating weaknesses.

NORM-REFRENCED Vs. CRITERION-REFRENCED TESTS

The definition for a criterion-referenced test (CRT) is:

A test which measures a student's performance according to a particular standard or criterion which has been agreed upon. The student must reach this level of performance to pass the test, and a student's score is therefore interpreted with reference to the criterion score, rather than to the scores of other students.

This is markedly different from the definition for a norm-referenced test (NRT) given in the same source:

a test which is designed to measure how the performance of a particular student or group of students compares with the performance of another student or group of students whose scores are given as the norm. A student's score is therefore interpreted with reference to the scores of other students or groups of students, rather than to an agreed criterion score.

The essential difference between these definitions is that the performance of each student on a CRT is compared to a particular standard called a criterion level (for example, if the acceptable percent of correct answers were set at 70 percent for passing, a student who answered 86 percent of the questions correctly would pass), whereas on an NRT a student's performance is compared to the performances of other students in whatever group has been designated as the norm (for example, regardless of the actual number of items correctly answered, if a student scored in the 84^th percentile, he or she performed better than 84 out of 100 students in the group as a whole).

In administering a CRT, the principal interest is in how much of the material on the test is known by the students. Hence the focus is on the percent of material known, that is, the percent of the questions that the student answered correctly in relation to the material taught in the course and in relationship to a previously established criterion level for passing.

In administering an NRT, the concerns are entirely different. Here, the focus is on how each student's performance relates to the scores of all the other students, not on the actual number (or percent) of questions that the student answered correctly.

In short, CRT's are designed, to examine the amount of material known by each individual student (usually in percent terms) while NRTs, examine the relationship of a given student's performance to the scores of all other students (usually in percentile or other standardized score terms).

The two types of tests also differ in:

(1) The kinds of things that they are used to measure,

(2) The purpose of the test

(3) The distributions of scores that will result

(4) The design of the test

(5) The students' knowledge of the test questions beforehand. Exploring each

Used to Measure

In general, NRTs are more suitable for measuring general abilities or proficiencies. Examples would include reading ability in Spanish or overall English language proficiency. CRTs, on the other hand, are better suited to giving precise information about individual performance on well-defined learning points.

Purpose of Testing

The purpose of an NRT must be to generate scores that spread the students out along a continuum of general abilities or proficiencies in such a way that differences among the individuals are reflected in the scores.

In contrast, the scores oh CRTs are viewed in absolute terms, that is, a student's performance is interpreted in terms of the amount, or percent, of material known by that student. Since the purpose of a CRT is to assess the amount of knowledge or material known by each individual student, the focus is on individuals rather than on distributions of scores. Nevertheless, as 1 will explain next, the distributions of scores for the two families of tests can be quite different in interesting ways.

Distribution of Scores

In other words, for an NRT to be effective softie students should score very low, and others very high, and the rest everywhere in between. Indeed, the way items for an NRT are generated, analyzed, selected, and refined will typically lead to a test that produces scores that fall into a normal distribution, or "bell curve.". For a CRT,

then, it is perfectly logical and acceptable to have a very homogeneous distribution of scores whether the test is given at the beginning or end of a period of instruction.

Test Design

NRT is likely to be relatively long and to be made up of a wide variety of different item types. An NRT usually consists of a few subtests on rather general language skills, for example, reading and listening comprehension, grammar, writing, and the like. These subtests will tend to be relatively long (30—50 items) and cover a wide variety of different test items.

In comparison, CRTs are much more likely to be made up of numerous, but shorter, subtests. Each of the subtests will usually represent a different instructional objective for the given course—with one subtest for each objective. For example, if a course has 12 instructional objectives, the CRT associated with that course might have 12 subtests

Students' Knowledge of Test Questions

Because of the general nature of what NRTs are testing and the usual wide variety of items, students rarely know in any detail what types of items to expect. The students might know what item formats they will encounter, for example, multiple-choice grammar items, but seldom will they be able to predict actual language points.

However, on a CRT, students should probably know exactly what language points will be tested, as well as what items types to expect. If the instructional objectives for a course are clearly stated and if those objectives are the focus on instruction, then the students should know what to expect in the test.

DIFFERENT INSTRUMENTS FOR DIFFERENT PURPOSES

Test Qualities	Type of Decision / test
Test Qualities	Proficiency	Placement	Achievement	diagnosis
Detail of information	Very general	General	specific	Very specific
Focus	General skills prerequisite to program entry	Learning points drawn from entire program	Instructional objectives of course or program	Instructional objectives of course or program
Purpose of decision	Compare individual overall groups/individuals with other	Find each student's appropriate level	Determine amount of learning with regard to program objectives	Inform students and teachers of objectives that still need work
Type of comparison	Comparison with other institutions	Comparisons within programs	Comparison to course or program objectives	Comparison to course or program objectives
When administered	Before entry or at the end of program	Beginning of program	End of courses	Beginning middle of courses
Interpretation	Spread of scores	Spread of scores	Degree to which objectives have been learned	Degree to which objectives have been learned
Type of test	NRT	NRT	CRT	CRT

ADOPT, DEVELOP, OR ADAPT

Many language tests are, or should be, situation specific. This is to say, a test can be very effective in one situation with one particular group of students and be virtually useless in another situation or with another group of students.

Other practical considerations include the initial and ongoing costs of the test and the quality of all of the materials provided. Is the test easy to administer? What about scoring? Is that reasonably easy given the type of test questions involved? Is the interpretation, of scores clearly explained with guidelines for presenting the scores to the teachers and students?

Clearly, then, a number of factors must be considered even when adopting an already published test for a program. Ideally, the program would have a resident expert,

someone who can help everyone else to make the right decisions. If no such expert is available, it may be advisable to read up on the topic yourself.

(TABLE)TEST EVALUATION CHECKLIST

A. General background information

1. Title

2. Author

3. Publisher and date of publication

4. Published reviews available

B. Theoretical orientation

1. Test family (norm-referenced -or-criterion-referenced

2. Purpose of decision (proficiency, placement, achievement, or diagnosis)

3. Language methodology orientation (approach and syllabus)

C. Practical orientation

1. Target population (age, level, nationality, language/dialect, educational background, and so forth)

2. Skills tested (for instance, reading, writing, listening, speaking, structure, vocabulary, pronunciation)

3. Number of subtests and separate scores

4. Type of items reflect appropriate techniques and exercises (receptive: true-false, multiple-choice, matching; productive: fill-in, short-response, essay, extended discourse task).

D. Test characteristics

1. Norms

a. Standardization sample

b, Type of standardized scores

2. Descriptive statistics (central tendency, dispersion, and item characteristics)

3. Reliability

a. Types of reliability procedures used

b. Degree of reliability for each procedure

4. Validity

a. Types of validity procedures used

b. Do you buy the above validity argument(s)?

5. Practicality

Cost of test booklets, cassette tapes, manual, answer sheets, scoring templates, scoring services, any other necessary test components
Quality of items listed immediately above {paper, printing, audio clarity, durability, and so forth)
Ease of administration(time required, proctor/examine ratio, proctor qualifications, equipment necessary, availability and quality of directions for administration, and so forth)
Ease of scoring (method of scoring, amount of training necessary, time per test, score conversion information, and so forth)
Ease of interpretation (quality of guidelines for the interpretation of
scores in terms of norms or other criteria)

DEVELOPING LANGUAGE TESTS

Proficiency, placement, achievement, and diagnostic tests can be developed and fitted to the specific goals of the program and to the specific population studying in it.

That might mean first developing achievement and diagnosis tests (which are

based entirely on the needs or the students and the objectives of the specific program), while temporarily adopting previously published proficiency and placement tests.

Later, a program-specific placement test could be developed so that the reasons, for separating students into levels in the program are related to the things that the students can learn while in those levels. It is rarely necessary or even useful to develop program-specific proficiency tests because of their interprogrammatic nature.

Naturally, all of these decisions are up to the teachers, administrators, and curriculum developers in the program in question.

ADAPTING LANGUAGE TESTS

The purpose of adapting a test to a specific situation will probably involve some variant of the following strategy:

Administer the test to the students in the program.
Select those items that appear to be doing a good job of spreading out the students for an NRT, or a good job of measuring the learning of the objectives with that population for a CRT.
Create a shorter, more efficient, revised version of the test that fits the ability levels of the specific population of students.
Create new items that function like those that were working well in order to have a test of sufficient length.

ORGANIZING AND USING TEST RESULTS

A checklist for successful testing :

A. Purposes of test

1. Clearly defined (theoretical and practical orientations)" :

2. Understood and agreed upon by staff

B. Test itself

C. Physical needs arranged

1. Adequate and quiet space

2. Enough time in that space for some flexibility

3. Clear scheduling

D. Pre-administration arrangements

1 .Students properly notified

2. Students signed up for test

3. Students given precise information (where "and when test will be, as well as what they should do to prepare and what they should bring with them, especially identification if required)

E. Administration

Adequate materials in hand (test booklets, answer sheets, cassette tapes, X pencils, scoring templates, and so forth) plus extras
All necessary equipment in hand and tested (cassette players, micro-phones, public address system, videotape players, blackboard, chalk, and so forth) with 'backups where appropriate
Proctors trained in their duties
All necessary information distributed to proctors (test directions, answers to obvious questions, schedule of who is to be where and when, and so forth)

F. Scoring