Levels of Integrity Data Analysis

Regardless of the observation system used, data should be able to be analyzed at the individual, team, and organizational levels. The job of the supervisor is to analyze the data at the individual and team levels. Individual analyses allow supervisors to determine the needs of an individual provider. Team analyses allow for information about how supervisory teams are performing and whether supervisors need additional support or if there are overarching areas where the entire team needs support. Organization-level analyses are typically done by organizational leaders.

Integrity data analysis helps determine the type of support or training is needed to help providers implement our programs/ treatment/ procedures with high levels of accuracy and utility.

Individual Level Analysis 

Analyzing the data at the individual level, allows supervisors to assess how a particular provider is performing and what programs are implemented well and poorly so appropriate support can be provided. Research suggests that providers' intention to turnover is dependent on the training they receive from their organization and the support they receive from their supervisors.

Case Example: Corinna

Corinna has been working at the organization for two months and is struggling across all of the areas assessed thus far. As you can imagine, if Corinna continues to perform low then they likely will not feel supported and stay with the organization for an extended period. That is unless the supervisor is using these data to allocate their support! Corinna's data would suggest that the supervisor needs to provide additional training or identify interventions to put in place to support Corinna’s performance improvement.

Case Example: Rikesh 

Rikesh has been working at the organization for months and his integrity scores are generally high (above 80%). However, there are some individual procedures that he struggles to implement with integrity. Doing individual-level analyses, for high performers as well, will reveal pinpointed areas that the supervisor could provide support for. 

Team Level Analysis

Analyzing the data at the team level, allows organizations to assess how a team is performing in relation to other teams and how a particular team is performing in specific areas. It is critical to use integrity data to evaluate what additional support supervisors may need. Recent research suggested that supervisors in our field are not regularly receiving training at their place of employment and that they want to expand on their skill sets and receive more support.

Case Example: Janae's Team 

Data for Jenae’s team reveals that her team is struggling to implement several procedures with integrity. Because there are multiple, this might be an indication that Janae, the supervisor, needs a little additional support. Specifically, the data for discrete trial teaching confirm that it is a team issue because multiple of Janae's team members are struggling with that program specifically. 

Case Example: Blanche's Team 

Data for Blanche’s team reveals that her team is performing high across multiple assessments and their performance exceeds the 80% standard. These data reveal that praise, incentives, or promotions could be used to continue to support Blanche in their efforts to have a high-performing team. It is important to note that although the team is performing high, observations and feedback should still be provided!

Organization Level Analysis

Organization-wide data allow for an analysis across all providers to inform organization-wide change initiatives. Some questions that one might ask themselves is how is the organization performing overall? What programs are implemented well or poorly across all providers?

This level of analysis will provide insights into how organizational leaders can course correct. Using the data one can determine the type of education, training, and or support needed to help support all providers in ensuring programs, treatments, and procedures are implemented with high levels of accuracy.

To be able to do these analyses, organizations must find a way to aggregate their individual integrity data. We recommend using standardized integrity evaluation tools so data aggregation is possible. We use Google Looker Studio for business intelligence, but other software is available and used by ABA organizations. 

Case Example: ABA Organization 1

Multiple assessments have been conducted multiple times where the average integrity is below 80%. Specifically, the data for functional communication training reveal that in July the average integrity across 11 providers was 36%. These data may suggest that additional data is necessary to determine if this is an outlier or could lead to an intervention being put in place. A risk to waiting is that it could continue to be low across months.

Which is what was observed for the natural environments training data. These data reveal that integrity averaged about 45% across two consecutive months! Since this pattern was observed for 14 providers, this would indicate that an organization-wide intervention is necessary.

Not only are these data used to inform interventions, but, they are also to determine whether an intervention was successful.

This organization's data for behaviors targeted for reduction revealed that an intervention was put in place at the end of August and integrity improved in September! That is, the intervention was successful, and continuing to measure data in this area will inform whether integrity levels continue to be high in the future.