AI Can Write the Questions. But Are They Any Good?

AI has made creating learning content ridiculously easy. Give it a topic, ask it to develop a course, and before you know it, you've got learning objectives, content, activities and a knowledge check. Usually with five multiple-choice questions thrown in for good measure.

And that's where I start to get a little twitchy.

Because AI seems to have a particular talent for producing one blindingly obvious correct answer and three alternatives that wouldn't fool a five-year-old.

Don't believe me?

Let's try one.

I'm guessing you chose A. Well done!

Although, let's face it, you probably didn't need to know anything about learning design to work that one out.

And therein lies the problem.

First, a confession

Multiple-choice questions aren't my favourite way of assessing understanding.

I'd much rather see learners making decisions in realistic situations, working through scenarios or demonstrating what they'd actually do in the workplace.

But MCQs have their place. They're quick, easy to mark and sometimes they're simply the most practical option. And when they're done well, they can tell us much more than whether someone remembers a fact.

So, if we're going to use them, let's make them count.

It's the wrong answers that interest me

The incorrect options in a multiple-choice question are called distractors.

And, in my experience, they're often the hardest part to write.

If three answers are obviously wrong, the learner doesn't really need to understand the content. They just need to spot the only sensible option.

They get a tick. The course records a pass. Everyone's happy.

But have they actually learnt anything?

Who knows?

I'd much rather give them alternatives that make them stop and think.

Perhaps two responses look equally reasonable at first glance.

Perhaps one response looks right at first glance, but misses an important detail.

Or perhaps an answer reflects a mistake someone might genuinely make in the workplace.

That's when things get interesting.

And yes, I've been known to spend more time writing the wrong answers than the right one.

There's a bit of brain science behind this

One of the things I find fascinating about learning is what happens when we have to work a little harder to remember something.

There's a well-researched concept called retrieval practice. Essentially, recalling information from memory can help us remember it for longer.

And psychologists have a rather wonderful term for certain challenges that support learning: desirable difficulties.

In other words, a little mental effort can be a good thing.

But there's a difference between making someone think and making something unnecessarily difficult. The challenge needs to come from thinking about the learning, not trying to decipher the question.

I'm not trying to catch learners out with tricky wording or answers that are so similar they need a magnifying glass to spot the difference.

I want them to recall something they've learnt. Apply it. Make a judgement.

Perhaps even go back and check something.

Because that thinking can be part of the learning.

So, how do I use AI to write MCQs?

I find AI particularly useful when I'm trying to come up with plausible wrong answers. Sometimes the correct answer is the easy bit. It's finding three alternatives that will genuinely make learners think that takes the work.

Depending on the question, I might:

• Ask AI to help me come up with a realistic scenario.

• Give it the correct answer and ask for three plausible alternatives.

• Write the answers myself and ask it to help tighten the wording.

Sometimes I'll go back and forth several times before I'm happy.

But the important learning design decisions are still mine. What am I trying to find out? What should the learner understand? What mistakes might someone reasonably make?

And my favourite test? Could someone who hadn't read the content still pick the correct answer?

If the answer's yes, those distractors probably need another look.

Right, let's see if you've been paying attention.

I've told you how I approach writing MCQs. Now let's see what you'd do.

You've asked AI to create a multiple-choice question for a workplace learning course. The correct answer is fine, but the three distractors are so obviously wrong that learners could probably guess their way through without understanding the content.

You decide to ask AI to have another go.

Got your answer? We'll come back to it.

Now, don't forget the feedback!

This is another area where I think we sometimes miss a trick.

We've gone to all that trouble to write plausible wrong answers, so why waste the opportunity to explain why they're wrong?

I'm not a huge fan of feedback that simply says Correct! or Incorrect. Try again.

Thanks. Very helpful.

If someone selects an incorrect response, I want them to understand why.

Maybe they've missed a small but important detail. Maybe they've misunderstood something. Or maybe their answer would have been perfectly reasonable in a different situation.

And if they got it right? Let's reinforce why.

After all, the learning doesn't have to stop when someone clicks Submit.

A little tip for Articulate Rise users

If you haven't discovered it yet, the feedback options in Rise now include By Choice.

Rather than giving everyone the same feedback, or simply distinguishing between correct and incorrect responses, you can provide feedback for each individual answer.

And it's really easy to use!

If you've already thought carefully about why someone might choose each distractor, you're already halfway towards writing useful feedback.

So, how did you go?

Remember that question I asked you earlier?

Let's see what you chose.

If you chose A...

You're thinking along the right lines.

Making the incorrect answers more similar to the correct one could certainly make learners stop and think.

But there's a difference between a meaningful distinction and a subtle difference in wording. If learners have to examine every word to work out which answer is correct, we might be testing their reading skills rather than their understanding.

A little challenge is good. Making things unnecessarily tricky? Not so much.

If you chose B...

That's the one I'd choose!

Asking AI to base the distractors on common misunderstandings or mistakes gives it something much more useful to work with.

Instead of producing three answers that simply look plausible, we're asking for alternatives that reflect how someone might genuinely misunderstand or misapply what they've learnt.

And this is where some of that learning science comes in. Learners need to retrieve what they know, consider the alternatives and make a decision based on their understanding.

That's the kind of thinking we're trying to encourage.

If you chose C...

I can see why you'd choose this one.

We certainly want learners to consider all the alternatives rather than immediately spotting the correct answer.

But there's a catch.

Making every answer equally convincing could leave us with several answers that are equally defensible. And that's not what we want.

A good MCQ needs one clearly defensible best answer. The distractors should be plausible, but there must be a meaningful reason why they're not the best choice.

Otherwise, we're testing whether learners can guess what the question writer had in mind!

If you chose D...

There's some merit in this approach too.

Using relevant terminology and detail from the course could make the incorrect answers sound more credible.

But adding technical language doesn't automatically make a distractor more effective.

If the extra detail doesn't reflect something learners might genuinely misunderstand, we're potentially adding complexity without improving what the question actually tests.

Remember those desirable difficulties we talked about earlier? The challenge needs to serve the learning, not simply make the question harder to read.

Did I make you think?

Did I make you think?

All four instructions could give you something better than those three ridiculous answers AI produced in the first place.

But better distractors don't necessarily mean better assessment.

The strongest instruction is the one that helps AI generate alternatives based on what learners might genuinely get wrong, and why.

And that's where your learning design expertise comes in. AI can generate the options, but it's still up to you to decide whether they're testing understanding or simply making learners work harder.

So, did you find yourself weighing up two answers? Perhaps you went back to check something I'd said earlier.

Good! That's exactly what I was hoping you'd do.

And if you chose something other than B, hopefully the feedback gave you something useful to take away.

One last thought

AI is brilliant at generating content.

But generating content and designing learning aren't quite the same thing.

A question can look perfectly professional, be beautifully worded and still tell you very little about whether someone has understood anything.

So next time AI hands you one blindingly obvious correct answer and three that barely deserve consideration, spend a little more time on those wrong answers.

Your learners deserve better.

A little further reading

If you're interested in the learning science behind some of this:

• Roediger, H. L. & Karpicke, J. D. (2006). Test-Enhanced Learning: Psychological Science and Practice.

• Bjork, R. A. & Bjork, E. L. (2011). Making Things Hard on Yourself, But in a Good Way.

• Butler, A. C., Karpicke, J. D. & Roediger, H. L. (2007). The Effect of Type and Timing of Feedback on Learning from Multiple-Choice Tests.