Submitted by Mehdi S, Iran

What procedures can we use to evaluate a published textbook?

Dr Richards responds:

Textbook evaluation can be divided into separate phases: pre-use (also known as pre-evaluation), during use (or in-use) and after use (or post-use).

Pre-evaluation: analysis
Most textbook evaluation schemes distinguish two essential stages that are necessary at the pre-evaluation phase: a description or analysis phase, and an interpretation or evaluation phase. In the first phase, the contents of the book have to be carefully described in terms of scope and sequence, organization, and the types of texts and exercises contained within. The analysis phase will involve identifying these kinds of information:

• Aims and objectives of the book.
• Level of the book.
• Skills addressed.
• Topics covered.
• Situations it is intended for.
• Target learners.
• Time required.
• Components.
• Number and length of units.
• Organization of units.

Pre-evaluation: evaluation
This stage of evaluation is more difficult since it involves subjective judgements, and these often differ from one person to another. For this reason, group evaluations are often useful. A number of checklists have been developed to assist at this stage of Pre-evaluation. However, checklists involve somewhat subjective categories and usually need to be adapted to reflect the particular book under consideration. In general, textbook evaluation addresses the following issues:

Goals: What does the book seek to achieve and how clearly are its learning outcomes identified?
Syllabus: What syllabus framework is the book based on? Is the syllabus adequate or would it need to be supplemented (e.g. through additional activities for grammar or pronunciation)?
Theoretical framework: What language-learning theory is the book based on? Does it present an informed understanding of any underlying theory?
Methodology: What methodology is the book based on? Is it pedagogically sound?
Language content: What kind of language does it contain and how authentic and relevant is the content? Is it an appropriate level of difficulty for the learners?
Other content: What topics and themes are covered and are they appropriate for the target learners?
Organization: Is the book well organized into units and lessons, and within lessons are the purposes of activities clearly identified? Do units have a coherent, consistent organization and do they gradually progress in difficulty throughout the book?
Teacher appeal: Does the book look easy to teach and is it self-contained, or would the teacher need to develop supplementary materials to use with it? Would it require special training or could it be used by teachers with limited experience, and by both native-speaker and non-native-speaker teachers?
Learner appeal: How engaging would it be for learners? How would they rate the design of the book (including the photos and illustrations), the topics and the kinds of activities included? Is the material clearly relevant to their perceived language-learning needs? Are self-study components included?
Ancilliaries: What other components does the book include, such as teacher’s book, workbook, tests, and digital and web-based support? Are all of these components published and available?
Price: Is the book affordable for the intended buyers?

When a group-evaluation process is used, all of the issues above and others specific to the teaching context can be discussed, and if several books are being considered, a consensus reached on the book that most suits teachers’ needs. The decision may not rest entirely on the book’s merits. For example, if students are known to use a certain coursebook in private high schools, the book may be rejected for use in private-language programmes that attract university students.

Evaluating during and after use
In-use evaluation focuses partially on the global needs of the institution: if testing is important, the comprehensive nature of the tests may be evaluated closely; if lab work is important, the pedagogical effectiveness and comprehensiveness of the online components may be evaluated in depth; if the school transitions students from a younger-learners programme to an adult programme, the ease of the transition from the coursebook for younger learners may be reviewed.

In terms of the classroom experience, however, and overall learner satisfaction, in-use evaluation focuses on how well the book functions in the classroom, and depends on monitoring the book whilst it is being used by collecting information from both teachers and students. Information collected can serve the following purposes:

• To provide feedback on how well the book works in practice and how effectively it achieves it aims.
• To document effective ways of using the textbook and assist other teachers in using it.
• To keep a record of adaptations that were made to the book.

This monitoring process may involve ongoing consultation with teachers to address issues that arise as the book is being used and to resolve problems that may occur. For example:

• Is there too much or too little material?
• Is it at the right level for students?
• What aspects of the book are proving least and most effective?
• What do teachers and students like most or least about the book?

Various approaches to monitoring the use of a book are possible:

• Observation: Classroom visits to see how teachers use the book and to find out how the book influences the quality of teaching and learning in the lesson.
• Record of use: Documentation of what parts of the book were used or not used and what adaptations or supplements were made to the book and why.
• Feedback sessions: Group meetings in which teachers discuss their experiences with the book.
• Written reports: The use of reflection sheets, or other forms of written feedback (e.g. blogs and online forums), in which teachers make brief notes about what worked well and what did not work well, or give suggestions on using the book.
• Teachers’ reviews: Written reviews by a individual or groups of teachers on their experiences with the book, and what they liked or didn’t like about it.
• Students’ reviews: Comments from students on their experiences with the book.

Post-use evaluation serves to provide information that will help decide if the book will continue to be used for future programmes.
Detailed information from textbook-evaluation processes, often conducted over a lengthy period, is a primary source of input when publishers decide to develop new editions of textbooks. Therefore, teachers may have a profound effect on the future direction of textbooks they are currently using.