Finding the same episode: reliability of corrective-feedback identification in naturalistic online tutoring
Résumé
Reliability in coded interaction research is usually reported for labels assigned to episodes that have already been identified. That practice leaves a prior question unanswered: would independent coders identify the same episodes in the first place? This brief report examines that question in naturalistic, automatically transcribed online English tutoring. Two coders independently screened the same predefined 1,004 learner turns from six de-identified adult lessons using a frozen corrective-feedback manual. This fixed-frame design evaluates agreement on turn-level episode presence; it does not estimate unrestricted segmentation or boundary placement in an unsegmented record. One coder identified 12 positive turns and the other 19; only three overlapped. Overall agreement was 97.5%, yet Cohen's kappa was .182 [95% turn-level bootstrap CI (–.011, .377)] and positive specific agreement was .194 (95% CI [.000, .385]). One shared positive had appeared as a worked example in the manual. Excluding that turn reduced kappa to .126 and positive specific agreement to .138, while leave-one-lesson-out analyses gave kappa values of .086–.240 and positive specific agreement of .095–.250. Classification agreement across the three shared positives was descriptive only (20/20, 11/20, and 18/20 matched fields); the perfect case was the worked example. The results show strong agreement on absence but weak reproducibility of the rare positive class. Corrective-feedback studies should separate identification from classification, state whether candidate units were predefined, publish the confusion matrix, and report class-specific alongside overall and chance-corrected statistics.
Citer ce document
Accès au document
Texte intégral en lecture en ligne, réservé aux abonnés SPHAERO et aux membres de l'institution. Se connecter
Voir l'article sur le site de la revueAuteur(s)
Statistiques
Consultations : 1
Téléchargements : 0