Creating 50,000 Aviation Exam Questions with AI

Aviation Infinity needed content. A lot of content.

The EASA ATPL exam covers fourteen subjects, each with thousands of possible questions. To build a comprehensive study platform, I needed tens of thousands of questions, each with correct answers, plausible distractors, and clear explanations.

Creating this content manually, one question at a time, would take years. Hiring a team of aviation subject matter experts would cost more than the product could generate in its first years of revenue.

I needed a scalable approach. The answer was AI-assisted generation with human expert review, a pipeline that let me produce content at scale while maintaining the accuracy that aviation education demands.

The Content Pipeline

The pipeline had four stages:

Stage 1: Source material analysis. The EASA syllabus defines exactly what topics each subject covers. I structured the source material (textbooks, regulatory documents, published guidance) into a topic hierarchy. Each topic became a content target with specific learning objectives.

Stage 2: Question generation. Using the structured source material, I generated candidate questions. In 2019, this meant template-based generation with variation. A topic like "altimeter settings" would produce templates: "At [altitude], with a QNH of [value], what is the pressure altitude?" with different values inserted to create unique questions.

Stage 3: Expert review. Every generated question was reviewed by someone with aviation knowledge, initially me and later by contracted aviation instructors. The review checked factual accuracy, question clarity, distractor plausibility, and alignment with the EASA syllabus.

Stage 4: Explanation writing. Each question received a detailed explanation of why the correct answer is correct and why the distractors are wrong. This stage was the most time-intensive and the most valuable, because the explanations are what students actually learn from.

The Quality Challenge

In aviation education, accuracy isn't optional. A wrong answer in a study platform doesn't just waste a student's time. It teaches them incorrect information that could affect their exam performance and, theoretically, their future decision-making as a pilot.

The quality bar is higher than general education:

Regulatory precision. A question about minimum descent altitudes must use the exact values specified in current regulations. An outdated value isn't just wrong. It's dangerously wrong if a student applies it operationally.

Distractor quality. Plausible wrong answers should be things a student might reasonably believe, not obviously incorrect options. "The speed of light" isn't a plausible answer to an aviation question. "The standard lapse rate minus 2 degrees" is, because it's close to correct but specifically wrong.

Explanation completeness. An explanation that says "B is correct" is useless. An explanation that says "B is correct because EASA OPS requires a fuel reserve of 30 minutes for IFR flights under specific conditions, while option A applies to VFR flights and option C applies to extended range operations" teaches the student the underlying principle.

The review process caught an initial error rate of about 15% in generated questions, not factual errors, but questions that were ambiguous, poorly worded, or had multiple defensible correct answers. This 15% couldn't reach students, which meant the review process couldn't be eliminated.

Scaling Content Without Scaling Teams

The key insight was that AI assistance reduced the work per question without eliminating human oversight.

Without AI: a subject matter expert writes a question from scratch (15-30 minutes per question including explanation).

With AI: the system generates a candidate question with draft explanation (seconds), and a reviewer validates and refines it (3-5 minutes per question).

The productivity improvement was roughly 5x, not by removing experts, but by reducing their work to judgment and refinement rather than creation from scratch.

This human-in-the-loop approach became my standard for AI content in regulated industries. The AI generates. The human validates. The final output carries the AI's speed and the human's accuracy.

The Feedback Loop

Once students started using the questions, a feedback loop emerged that improved content quality over time:

Difficulty calibration. Questions that every student answered correctly were too easy. Questions that no student answered correctly were too hard (or ambiguous). The data revealed which questions were at the right difficulty level for learning.

Error detection. When multiple students reported the same question as incorrect, it flagged a potential content error. Student reports became a distributed quality assurance system.

Explanation effectiveness. By tracking whether students who read the explanation performed better on related questions later, I could measure which explanations actually taught and which didn't.

This data-driven content improvement was only possible at scale. With a hundred questions, the data is too sparse. With tens of thousands of questions and thousands of students, the statistical signal is strong enough to drive meaningful improvements.

Content creation at scale isn't a one-time effort. It's an ongoing system: generate, review, publish, measure, improve. Each cycle makes the content better, and the compounding effect over years creates a content asset that's difficult to replicate.