Observing Teaching Behavior Using the International Comparative Analysis of Learning and Teaching Measure Across Countries: Is There Measurement Invariance?

Objectives

Measurement invariance (MI) is a necessary condition when comparisons of scores across countries is aimed for. In this study, we investigated MI of the International Comparative Analysis of Learning and Teaching (ICALT, Van de Grift et al., 2014) instrument for measuring teaching behavior and we subsequently compared effective teaching behavior across 10 countries.
Theoretical framework

We used the effective teaching behaviour model (ETBM) of Van de Grift (2007) as the theoretical framework for studying teaching behaviour. The ETBM is based on reviews of evidence-based teaching effectiveness research (e.g., Creemers & Kyriakides, 2008). The model includes six observable domains of teaching behaviour that have been proven to affect students’ learning and outcomes: Learning Climate, Classroom Management, Clarity of Instruction, Activating Teaching, Differentiated Instruction, and Teaching Learning Strategies. We used structural equation modelling (SEM) to study factor structure and measurement invariance.

Method and data analysis

The study included 3,583 teachers from 10 countries: The Netherlands (Nteacher = 606), Indonesia (Nteacher = 335), South Africa (Nteacher = 304), South Korea (Nteacher = 581), Hong Kong-China (Nteacher = 218), Spain (Nteacher = 344), The United Kingdom (Nteacher = 153) and the United States (Nteacher = 320). The ICALT instrument (Van de Grift et al., 2014) was used for measuring teaching behaviour (Table 1).

The instrument was translated and back-translated from English to the participating countries’ languages following the international guideline (Hambleton, 1994). Classroom observers were trained by expert trainers. Inter-rater reliability ranged from 70% to 96%.

The measurement model was tested in each country separately using Categorical Confirmatory Factor Analysis (C-CFA), followed by Categorical Multi-group Confirmatory Factor Analysis (C-MGCFA). MPlus 8.1 (Muthén & Muthén, 2017) was used. Configural, metric, and scalar levels of MI were tested (Putnik & Bornstein, 2016).

Findings and conclusion

The full measurement model of C-CFA, shows reasonable model fit for most countries, except for Hong Kong-China, Spain and the USA (Table 2). The USA data yielded the worst fit (excluded for C-MGCFA).

Results of C-MGCFA with all 32 items for nine countries confirmed configural invariance only (Table 3). A closer inspection of modification indices revealed that items 17 and 22 were problematic in multiple countries. After deleting these two items, configural, partial metric, and scalar invariance was reasonably confirmed.

Using the Netherlands as reference group (Table 4), South Korean teachers outperformed on the six domains of teaching behaviour. Mongolian and South African teachers were rated lower on the basic skills, but higher on the complex skills. English and Spanish teachers were rated higher on all domains except learning climate. Indonesian, Pakistan, and Hong Kong – Chinese teachers were rated lower on all domains, except teaching learning strategies.

Significance

This study contributes to the measurement field by applying the C-MGCFA technique based on the SEM framework for studying measurement invariance of the ICALT scales across countries. It paves the way for investigating the non-invariance items more specifically in each country on its potential sources and causes to inform further improvement of the measure.