BST Insights

I often tell my students that we are behavior detectives. When faced with a behavior, we have to put on our detective caps and figure out why it’s happening. Two tools we have as behavior analysts are treatment fidelity and interobserver agreement (IOA). These tools are often confused, but it’s important to remember they are different. Treatment fidelity is the accuracy with which an intervention is implemented as intended. Treatment fidelity can be measured in various ways, including self-report, permanent products, and direct observation. Interobserver agreement is a measure of the reliability of the data collected by two or more observers at the same time. It tells us how believable the data are. IOA can be measured in a variety of ways as well, but it depends on the type of data collected. (see Cooper, Heron, and Heward’s [2020] fifth chapter for a refresher on IOA methods). 

Treatment fidelity and IOA go hand-in-hand. They give us information about the programs we are running. Without this information, it’s as if we are walking blindfolded through a field. If a client is not making progress, the first question we should ask ourselves is, “Is the behavior technician implementing the program the way we’ve designed it?” If they’re not, then that tells us we may have to do some additional training on how to run the program. If the behavior technician’s treatment fidelity score is high, indicating they’re running the program as expected, then we need to assess the target behavior’s operational definition and measurement system. If two observers collect data simultaneously on the same target behavior and the score is low, then it tells us that we may need to revise our operational definition or provide booster training. While IOA is an important metric for evaluating data collection reliability, it is the fidelity of treatment implementation that ultimately determines an intervention's success. If our fidelity is low, then why check IOA?

In our clinical practice, it is important to make sure that we are assessing how well behavior technicians and behavior analysts implement procedures. The greater the fidelity of the intervention, the better the outcome for the client. Research shows that treatment fidelity scores of 80% or greater lend themselves to better outcomes for the client. Because we don’t have endless amounts of time with our clients, we need to make sure we’re capitalizing on the time we do have with them and providing effective treatments. When we collect treatment fidelity data, we also verify that our programs are what caused the behavioral changes we see. Without these data, we cannot say with certainty if it was our treatment that caused the skill acquisition or behavior reduction to occur. Collecting treatment fidelity data also helps to avoid any treatment drift that may occur and provides feedback to a supervisor if additional training is warranted. 

When we collect IOA data, the higher the IOA score, the more trustworthy our data are. We can say with more certainty that our operational definitions are clear, our data collectors are well trained, variability in the data isn’t due to having multiple data collectors/behavior technicians, and we don’t have observer drift. Low IOA scores indicate an issue that we need to fix. 

As a behavior analyst supervisor, we can easily collect treatment fidelity and IOA data during our routine observations. There are a variety of electronic resources now available that can help make this data collection easier (e.g., BSTperform for collecting treatment fidelity; other Data Collection software for collecting IOA). It’s vital that we take a few minutes during observations to collect these data so that we make sure we’re supporting our technicians and clients to the best of our abilities. 

In our journey as behavior detectives, the discerning use of treatment fidelity and IOA sharpens our clinical effectiveness, ensuring our interventions are as impactful as intended. Leveraging technology enhances our ability to deliver precise, evidence-based therapy, directly benefiting our clients with quality care. By prioritizing these metrics, we uphold the integrity of our practice and drive meaningful progress, setting a new standard for excellence in ABA therapy.

Behavior Science Technology is committed to furthering the field through research. Much of our research efforts focus on effective supervision practices, treatment fidelity processes, and client outcomes. The overarching goal is to ensure quality services are delivered to consumers of service and that providers receive the best support they can. Below we outline our research collaborations! 

In our quest to tackle the high turnover rates in the ABA industry, our data-driven investigation led to a pivotal discovery: the '4-80 rule.' This rule indicates that a minimum of 4 supervisory observations per month, with a mean treatment fidelity score of 80% or higher, greatly enhances technician retention and overall industry quality.

We have data from some of our long-standing customers who are now delivering high-quality services because they have a process in place at their organization to collect treatment fidelity data. 

We have data from organizations that use our software to show there is a wide variability and inconsistency in the quality of services being provided to clients. Ultimately, we do this work because we care about those that we serve and we strive to provide the highest quality service possible. Without having access to treatment fidelity data and doing regular analysis, it is unknown whether the quality of services provided are in fact optimal. 

Regardless of the observation system used, data should be able to be analyzed at the individual, team, and organizational levels. The job of the supervisor is to analyze the data at the individual and team levels. Individual analyses allow supervisors to determine the needs of an individual provider. Team analyses allow for information about how supervisory teams are performing and whether supervisors need additional support or if there are overarching areas where the entire team needs support. Organization-level analyses are typically done by organizational leaders.

Having spent the past 30+ years in new product development, it's somewhat disheartening that I have to once again discuss the displacement of "business as usual" by new technology. Throughout history, we've witnessed the transformative power of innovation, yet the resistance to change remains deeply rooted in human nature. There's a long-standing statement that holds true: "You will not be replaced by (FILL IN TECH), but your company will be replaced by those using (SAME TECH)." This was evident with the advent of the internet, and it holds true with the rise of AI. As I reflect on my experiences, I wanted to create a piece that transcends time, aiming to address the enduring nature of resistance to change, even long after I am gone or have finally given up.

Earlier this year, we embarked on an exciting collaboration with our colleagues at LEARN Behavioral to understand what BCBA supervisors are doing in practice. The mission of this research is clear: We aimed to make procedural integrity not just a standard, but a seamless and natural aspect of every BCBA's practice. To do this, though, we need to know how actively BCBAs are engaging in activities centered around procedural integrity AND if there is a shortfall in engagement, what's holding them back? What challenges are they encountering in their daily practice that might deter them from ensuring procedural integrity? We administered a comprehensive survey to gather this information.

As a life-long Mets fan, I’ve watched their winning percentage fluctuate from a dismal 30% to an outstanding 67%. The outcome was never a surprise; disappointing and exciting, but never a surprise. Why?