Using an AI tool and engineering an AI application are different tasks
Imagine a fictional assistant answering questions about an organisation’s public handbook. A user interface receives a question, software retrieves relevant material and a model proposes an answer. The engineering responsibility includes access, error handling, evidence and what the application does when the handbook does not contain the answer.
An AI engineer may work with software developers, data specialists and product owners. Building a larger model is not the only kind of work in this field. Integration choices and evaluation can matter even when the model is accessed through an API rather than trained from scratch.
Make data and software foundations visible
Python, data handling, APIs and basic programming make it easier to understand inputs, outputs and failure cases. Model selection and evaluation add questions about the intended task and acceptable errors. TWA’s existing direction includes these foundations, embeddings, retrieval-augmented generation, AI workflows and deployment/monitoring concepts.
Microsoft’s RAG documentation explains the role of retrieving grounding material for a generative application. Retrieval is not proof that a generated answer is correct. Google’s evaluation guidance also emphasises assessing responses against criteria. These are industry references, not a guarantee that TWA includes a particular provider, model or evaluation service.
Help a library assistant answer from evidence.
A visitor asks when a fictional library closes today. An old page says 17:00, today’s notice says 15:00 and a café menu mentions 14:00.
Your role: Choose useful evidence, write a bounded answer and make its source checkable.
What you’ll learn: Use current source information to constrain and check an assistant’s answer.
No account needed. Nothing is sent or saved. You can restart at any point.
Read the full guided story without the interactive controls
You can follow the same lesson here without JavaScript.
See why a plausible answer can fail
Try the archived page and compare its date with the question about today.
What happens and why: The archive says 17:00 for last year. Repeating it would not establish today’s hours. A fluent answer can still rely on stale information.
Retrieve evidence that answers the question
Choose documents to bring into the answer workspace. Include today’s library notice.
What happens and why: Retrieval means bringing relevant information into context. Today’s library notice supports 15:00; a café menu is about a different service and the archive is superseded.
Draft only what the evidence supports
Write the closing time shown by the current notice. Do not infer other opening arrangements.
What happens and why: The supported time is 15:00 today for maintenance. Validation compares an answer with the source; it does not require inventing extra detail.
Make verification part of the answer
Attach the current notice as a citation and require the assistant to abstain when current evidence is absent.
What happens and why: A citation lets the reader inspect support. Abstention means saying the evidence is insufficient instead of inventing an answer. No model or real library is connected.
Takeaway: Retrieve relevant, current evidence, limit the answer to its support and make verification possible. If the evidence is missing, acknowledge that limit.
Where this is useful: Useful when designing document assistants, help centres and evidence-based AI workflows.
A fictional educational example, not a confirmed TWA assignment, assessment or learner outcome. Simplified to explore one idea; real situations need context and authorised procedures.
Optional deeper exploration: open the original technical workspace
Want to explore the technical model?
The guided story above introduces the idea. This original workspace lets you try actions in different orders. It has its own reset and does not inherit your story choices.
- Retrieval
- Bringing relevant source information into the answer context.
- Validation
- Checking an answer against evidence and requirements.
- Citation
- A reference that lets a reader inspect the supporting source.
- Abstention
- Declining to assert an answer when support is insufficient.
Evidence before an answer.
Learning objective: Retrieve relevant current evidence and validate a bounded response.
A fictional library assistant is asked when the library closes today. No model or real library is connected.
Evidence desk
- Archived page
- Last year: closes 17:00
- Current notice
- Today: closes 15:00 for maintenance
- Unrelated document
- Café menu: lunch until 14:00
Your workspace
Worked explanation & takeaway — available without JavaScript
Retrieve the current library notice, cite it, and restrict the answer to its scope. Similar-looking or stale evidence can produce a confident wrong answer. If current evidence is excluded, abstain and request a current source rather than invent a closing time.
Fictional educational model. No live system, personal assessment or confirmed TWA assignment. All actions run locally in your browser.
What if the handbook has no answer?
No model is connected. The example shows an application decision that should be designed and tested.
- Question
Receive a task within a stated scope.
- Retrieve
Look for relevant permitted material.
- Respond
Generate an answer or an honest limitation.
- Evaluate
Compare the result with the expected behaviour.
Use the permitted material, preserve its context and check whether the response actually answers the question. A citation alone is not a correctness test.
Read both scenarios and explanations
Relevant evidence is available
Use the permitted material, preserve its context and check whether the response actually answers the question. A citation alone is not a correctness test.
The answer is missing
Say that the available material does not establish the answer and provide an appropriate human follow-up. Do not invent a policy to keep the conversation flowing.
An educational illustration using fictional situations. It is not a confirmed TWA assignment, live system, assessment or promise of a particular tool.
Define the missing-answer behaviour before the impressive demo
For the fictional handbook assistant, list questions the material can answer and questions it cannot. A useful test case records the question, relevant source, expected behaviour and observed result. An answer that sounds confident but invents a policy should fail that case even if its grammar is excellent.
Consider the full path: retrieving the wrong passage, retrieving nothing, a model or network failure, and a request the application is not authorised to perform. A safe-looking prompt does not replace access controls in ordinary application code. Keep private information out of a prototype unless the access and data-handling arrangements are approved.
A public-handbook assistant evaluation pack
Prototype or design an assistant using a short fictional handbook and sample questions. Keep the material non-sensitive and make missing answers explicit.
What you could document
- A system/data-access diagram
- A small labelled question set including unanswerable cases
- A prototype or application-flow description
- An evaluation report with failures, changes and limitations
What makes the explanation useful?
A reviewer should be able to distinguish retrieved evidence from a generated claim and reproduce why a case passed or failed. Fluency alone is not the success criterion.
The learning plan determines actual activities and assessment. These deliverables are examples, not contractual curriculum outcomes.
Improve the system through evidence
Compare a small baseline with a changed retrieval rule or prompt using the same cases. Document both improvements and regressions. A single successful example is not evidence that every future question will work. Include a way for a person to review a result and understand its limitations.
Deeper directions can involve applied AI development, machine-learning engineering or platform operation, depending on foundations and scope. Discuss your programming and data experience with TWA. AI Engineering has enquiry-based fees; it is not automatically covered by the fixed data or cloud engineering prices.