Evidence-based edtech

Question

There's a lot of edtech that promises results and not much that shows its work. We're collecting questions for an Educast 3000 podcast episode on how to tell the difference, and we'd like yours.

What are your questions about evidence-based edtech?

Ask us anything: how to read a vendor's efficacy claims, what "evidence-based" should actually mean before you buy, or how to run a pilot that tells you something real. Post your question in the thread.

If we use your question on the show, we'll tag you so you can catch the answer.

Your hosts: @Melissa Loble, Chief Academic Officer at Instructure and a learning futurist with 25+ years across K-12, higher ed, and workforce development, and @Ryan Lufkin, VP of Global Academic Strategy and a two-decade edtech veteran recently named to EdTech Digest's Top 100 Influencers in EdTech.

Jeff_F · Answer

And this third set of questions focuses on how institutions evaluate products before purchase, particularly whether procurement processes themselves are evidence based.

As institutions evaluate new edtech solutions, they are often bound by formal procurement processes. Many vendors make compelling claims, but comparing products objectively can be challenging when pricing, licensing, and feature sets vary widely. Here are a few questions that come to mind:

* What evidence should institutions request during an RFP process beyond customer testimonials and case studies? Should vendors be expected to provide independent research, outcome data, retention metrics, or other forms of evidence? What constitutes sufficient proof?
* How should institutions weigh feature comparisons against evidence of educational impact? A vendor may have more features than a competitor, but is there evidence that those features improve learning outcomes, faculty efficiency, student retention, or other meaningful measures?
* When comparing similar products, how can institutions distinguish genuinely impactful features from "checkbox features" that look impressive in demonstrations but see little actual adoption?
* How much transparency should vendors provide regarding product usage at existing institutions? For example, should institutions ask what percentage of licensed users actively engage with the tool after the first year?
* Which pricing model best aligns vendor incentives with institutional success? Per FTE, per IPEDS enrollment, institution-wide licensing, usage-based pricing, seat licenses, consumption models, or some combination?
* How should procurement teams evaluate low-adoption but high-cost products? If only a small percentage of faculty or students use a tool, what evidence should justify maintaining the investment?
* What makes an edtech pilot meaningful rather than merely a product demonstration? What outcomes, comparison groups, usage thresholds, and success criteria should be established before the pilot begins?
* Should institutions require vendors to define in advance what pilot results would constitute success or failure? If so, what measures are most credible?
* How long should a pilot run before meaningful conclusions can be drawn? Is a single semester sufficient, or should institutions look for evidence across multiple terms and instructional contexts?
* If two products produce similar outcomes, how should institutions evaluate differences in total cost of ownership, implementation effort, support requirements, accessibility, and long-term sustainability?
* If a vendor's pricing increases in direct proportion to institutional size (FTE or IPEDS enrollment), what evidence should they provide that a larger institution receives proportionally greater value than a smaller institution using the exact same product?

The goal here is to encourage both vendors and institutions to think critically about whether pricing models are rooted in demonstrated value or simply purchasing power.

For me, the most interesting questions are those that move beyond "Does it work?" and into "How do we know it works well enough to justify the investment?" Such questions seem to be the type that instructional support teams, procurement groups, and academic leaders wrestle with.

PatriciaRoy · Answer

Jeff —

These are fantastic questions. Thank you for sharing. I will be sharing with my team.

Jeff_F · Answer

I thought I'd add a few more -

The questions below are intended to encourage thoughtful discussion about the evidence behind publisher courseware and textbook integrations. The goal is not to challenge these products for the sake of criticism, but to apply healthy skepticism to a long standing and often expensive component of higher education. If we expect evidence-based decision making in other areas of edtech, it is reasonable to ask the same questions of textbook platforms and LTI integrations that can cost students well over $100 per course.

Based on the discussion prompt about evidence-based edtech, I would push textbook publishers to move beyond marketing claims and provide evidence that their expensive textbook-plus-LTI ecosystems actually improve learning outcomes. Here are five questions that could stimulate a meaningful discussion:

* What evidence demonstrates that students using your textbook and LTI integration achieve better learning outcomes than students using lower-cost alternatives such as OER materials paired with standard Canvas assessments?

Many publishers highlight engagement metrics and platform usage, but what evidence shows superior learning, retention, or transfer of knowledge?
* Your products often add $100 to $180+ in costs for students beyond tuition and fees. What independent, peer-reviewed research demonstrates that the additional educational value justifies that financial burden?

This question challenges publishers to connect price directly to measurable student benefit rather than convenience or feature sets.
* How do you distinguish the educational impact of the textbook content itself from the impact of course design, instructor quality, and required completion of graded LTI activities?

If students perform better, how do you know the improvement came from the product rather than instructional practices surrounding it?
* If an institution replaced your integrated courseware with an openly licensed textbook, Canvas quizzes, and instructor-created assignments, what evidence suggests student performance would decline?

This is a particularly challenging question because it requires publishers to demonstrate value beyond features that many LMS platforms already provide.
* What evidence shows that students retain knowledge and skills after the course ends, rather than simply earning higher course grades while interacting with your platform's adaptive learning tools, hints, and automated feedback?

Improved course grades are not necessarily evidence of deeper or more durable learning.

These questions focus on causation rather than correlation, learning outcomes rather than engagement metrics, and value relative to cost, which are often the hardest areas for edtech vendors to substantiate.