• By Dartington SRU
  • Posted on Thursday 09th April, 2015

In implementation, “how well” is as important as “how much”

Most studies of implementation look at whether facilitators cover the material in the programme manual. But they tend to neglect the role of competence in delivery. A recent study of a school anti-bullying programme found that teachers who taught with warmth and praise, and who covered the material clearly, were also the ones who got the best results.

Bullying prevention programmes in schools typically have limited impact. However, when programmes monitor implementation, the effect is doubled. One reason that programmes fail to deliver results, then, may be the level and quality of implementation – together known as “treatment fidelity.”

A study of an anti-bullying programme in two US middle schools aimed to break down treatment fidelity into four component parts, addressing different aspects of facilitator behaviours. Researchers then examined how these four types of behaviours related to pupils’ responsiveness.

All four behaviours were linked to pupil engagement and willingness to follow the anti-bullying rules.

It made a great deal of difference whether teachers followed 10 specified behaviours, such as praising pupils’ engagement and showing warmth and interest toward the pupils.

However, when the other behaviours were taken into account, it mattered relatively little whether teachers followed the pre-agreed class meeting outlines. In other words, it seems that it mattered “how well” the programme was delivered, independent of “how much” was delivered.

The Olweus Bullying Prevention Programme is a violence prevention programme that aims to enhance the school environment. It has three components: individual, classroom and school. This study focused on the central component, the classroom level. In weekly sessions, teachers cover a range of topics such as coping with anger, problem-solving, and stress management. The programme uses role-play, small group activities, and discussions.

Of 22 weekly meetings in each school year, the first six were based on the programme manual and the rest were developed in collaboration with the school’s Bullying Prevention Coordination Committee, taking into consideration pupil and teacher feedback.

In this study, researchers observed 44 teachers in two middle schools in the southeastern US. Both schools involved pupils in grades 6, 7, and 8 (approximately ages 12-14).

Most studies of fidelity focus on adherence (the extent to which the programme was delivered according to design). In this study, the authors also looked at quality (how competently the intervention was delivered).

They divided adherence into two categories. “Procedural adherence” was the degree to which teachers conducted the programme according to the pre-agreed outline of the meeting. “Instructional adherence” was a measure of whether teachers used 10 specified behaviours, such as demonstrating warmth toward pupils.

They also divided competence into two categories. “Procedural competence” was how well teachers followed the meeting outline, and how clearly they explained the programme content. “Instructional competence” was a measure of how well teachers delivered the 10 specified behaviours.

They then examined the relationships between each of the four aspects of facilitator behaviour and pupil responsiveness. Responsiveness included both the level of pupils’ engagement and their willingness to abide by the intervention.

The authors found that the amount of specific instructional behaviours teachers followed (instructional adherence) predicted pupil engagement more than how much of the meeting outline they delivered (procedural adherence). For example, if the teacher extensively demonstrated warmth and interest toward and involvement with the pupils, pupil engagement would be higher as compared to when the teacher extensively explained the purposes of class meetings.

On the other hand, how well teachers followed the meeting outline (procedural competence) predicted pupil involvement more than how well teachers delivered instructional behaviour (instructional competence).

Overall, the researchers point out, delivery with competence gives a boost to programme results, over and above the effect of delivery with adherence. It seems intuitively clear that quality should matter on top of quantity, and this study helps to show that this intuition is correct.

This study demonstrates that the quality of facilitators’ delivery is important in engaging participants and delivering results. Therefore, strategies to improve teachers’ practices, such as directed consultation and practice-based coaching, may improve the results of class-based interventions.

Future studies should also consider how pupil engagement relates to teacher-pupil relationships, pupil behaviour, and other outcomes such as levels of bullying. This is important because participant responsiveness is an aspect of implementation, and not the final goal.

Goncy, E. A., Sutherland, K. S., Farrell, A. D., Sullivan, T. N., & Doyle, S. T. (2014). Measuring teacher implementation in delivery of a bullying prevention programme: the impact of instructional and procedural adherence and competence on pupil responsiveness. Prevention Science, DOI 10.1007/s11121-014-0508-9

Return to Features