Does It Work: Five Checks for Efficacy

We alll search for educational technology that will help us achieve the learning outcomes established for our students or that allows us to add a new tool to our box that we can then share with other instructors and educators who are likewise looking for the most effective learning technologies.

We also hear many claims about learning apps and web-based edtech that say they enable educators to achieve their student learning outcomes. Do they actually do what they say they will? We’d like to know that before we invest our time in gaining familiarity with the technology, and it’s rare to find reviews that might provide some guidance.

In the article “Five Things About Efficacy That Should Be Intuitive But Are Not” Patricia Gomes shares questions used by experts to establish the efficacy of educational technology tools. Here’s a summary:

• Does the evaluation begins with having a clear understanding of what the outcomes are and how this technology will be used to achieve them? Start with the purpose behind the desire to adopt the educational technology
• Is the “active ingredient” claimed for the product grounded in sound pedagogic principle or does it seem that the product’s potential is based on questionable evidence? Be able to identify what the active ingredient is and why it’s been selected to power this product.
• Are there unique circumstances in which this edtech is more likely to work, with some students or for some subject matter, but not necessarily for all? It helps to be clear about the situations in which the technology is most likely to work and when it is likely to fail – and how that figures into your use of that technology.
• Will you be able to allot enough time to the use of this educational technology to really give it a chance to show what it can do? Just using the technology once or twice may be insufficient for evaluating its capacity to meet the intended purpose.
• Is there going to be a systematic approach to the collection of evidence, positive and negative, over time? It may be that some technology is required that supports the measurement process.

I like the idea of having a methodical approach to establishing the outcomes, purpose, time and measurement of an evaluation approach that can guide us each time we want to examine a new educational technology. But with the edtech world advancing so rapidly, with new technology being released constantly, I wonder how likely it is Blended Librarians will be able to take a rigorous approach to the adoption of edtech. Given that many Blended Librarians are also responsible for traditional library responsibilities, it may be a real challenge to allot the necessary time to assess the edtech efficacy.

As with other resources that librarians evaluation, it may be that we need to bring to the process a combination of gut instinct about what works and what doesn’t, along with sharing the evaluation with others to get their feedback. Does it work for them? Is it intuitive? It is likely to support the achievement of learning outcomes?

If there are more people experimenting with a specific educational technology, that can be a valuable way to gain the necessary feedback to at least make a first-cut decision. Then, with a greater level of confidence in that technology and its potential efficacy, a more rigorous evaluation process could be applied.