AI Can Be Confident and Wrong

AI confidence is a feature of presentation, not evidence of correctness. Generated output can contain false citations, nonexistent APIs, code that works only for the example, hidden assumptions, or unnecessary complexity.

Break the response into claims. Verify factual claims in primary sources and technical behavior with official documentation and reproducible tests. Mark unsupported claims as uncertain or reject them.

Confabulation can sound specific

NIST’s Generative AI Profile uses confabulation for erroneous or false content that a generative AI system presents confidently. The report notes that output can also contradict the prompt or earlier output.

OpenAI similarly states in its current explanation of language-model hallucinations that plausible but false statements remain a challenge.

These sources establish a known risk. They do not imply that every answer is wrong or that one vendor’s behavior represents every system. Each material claim still needs case-specific evidence.

Failure pattern 1: invented citations

A generated citation may contain:

• a real author with a nonexistent title;
• a real journal with wrong volume or pages;
• a real paper that does not support the claim;
• a fabricated DOI or URL; or
• a quotation absent from the source.

Verify in this order:

1. locate the source through the publisher, DOI registry, library, or official repository;
2. match authors, title, date, and identifier;
3. open the source;
4. find the relevant passage or result;
5. compare the source’s conditions and uncertainty with the AI claim; and
6. cite the source you actually read.

A search result that repeats the title does not show that the source supports the claim. An AI-generated quotation should not be used until checked against the original.

Failure pattern 2: invented or mismatched APIs

Generated code may name:

• a method that does not exist;
• an option from another tool;
• an API removed in the installed version;
• parameters in the wrong order;
• a return type that the real API does not provide; or
• a package with a plausible but incorrect name.

Check the official documentation for the exact product and version. Inspect release notes when behavior changed. Write the smallest harmless call that tests the claimed interface.

Do not install an unknown package merely because generated code imports it. Confirm the authoritative project and course requirement first.

Failure pattern 3: sample-only code

Code can produce the expected sample output while failing for:

• empty input;
• one item;
• duplicated values;
• unexpected characters or encoding;
• large input;
• invalid types;
• alternate ordering; or
• a different environment.

Derive cases from the specification rather than asking the AI which tests would make its own code look correct.

For a function that returns the largest value, one positive sample does not test all-negative input, empty input, or ties. Record which behaviors the specification defines and which remain unresolved.

Failure pattern 4: hidden assumptions

An answer may silently assume:

• the operating system and shell;
• a current directory;
• internet access;
• a software version;
• administrator privileges;
• clean, complete input;
• a specific unit or timezone;
• permission to share data; or
• an AI-use policy.

Turn assumptions into explicit questions:

Claim:
Required assumptions:
Evidence that each assumption holds:
What changes if one assumption is false:

If the answer recommends a command, resolve paths, variables, privilege, and network destinations before execution.

Failure pattern 5: unnecessary complexity

Generated output may introduce frameworks, dependencies, design patterns, or abstractions not required by the task. Complexity can make an answer appear advanced while increasing:

• installation and update obligations;
• failure modes;
• code to review;
• security surface;
• maintenance cost; and
• distance from the course concept.

Ask:

• Which requirement justifies this component?
• Can the current language or tool solve the task directly?
• Does removing it change the observable outcome?
• Can the student explain and maintain it?

Prefer the smallest solution that meets the specification. Simplicity is not proof of correctness, but unjustified complexity is a review finding.

Write an evidence record

Audit one response in a short record titled Evidence audit — [response or task]. Write one paragraph for each material claim, stating:

• the claim and whether it is a citation, API, behavior, assumption, or recommendation;
• the evidence required, such as an original paper, versioned official documentation, a small test, or a requirement mapping; and
• the result: supported, corrected, uncertain, rejected, or not material.

Do not fill in a Markdown table. Group related claims in a paragraph and mark missing information uncertain rather than leaving a row that looks unfinished.

Classify each result:

• supported: evidence matches the claim and conditions;
• corrected: the evidence supports a narrower or different claim;
• uncertain: adequate evidence is unavailable;
• rejected: evidence contradicts the claim; or
• not material: the claim does not affect the decision or result.

Do not convert uncertain into supported because the response sounds sure.

Separate sourced facts from recommendations

This article contains two kinds of statement:

• Sourced fact: NIST documents confabulation as a generative-AI risk.
• Recommendation: Students should decompose material claims and verify them with primary sources or tests.

The recommendation is a practical method. It does not claim to eliminate every error. Sources can be misread, tests can omit cases, and official documentation can be outdated relative to a local version. Record those limitations.

High-stakes medical, legal, financial, safety, or security decisions require domain-appropriate authoritative sources and qualified human review. Do not use an AI confidence signal as a substitute.

Common mistakes

• Checking whether the citation exists but not what it says. Read the source.
• Trusting a plausible API name. Verify exact versioned documentation.
• Running only the sample. Derive boundary and invalid cases.
• Leaving assumptions implicit. State and test them.
• Keeping complexity because it looks professional. Map it to requirements.
• Treating every AI statement as false. Classify using evidence.
• Asking the same AI to certify its answer. Use independent primary sources and observable tests.

Do this now

Choose one permitted AI response containing a factual or technical claim. Build the evidence table. Verify at least one citation or API in a primary source and one behavior with a reproducible test.

Log what you learned

The claim-verification record is the learning log. Add each source or test result only after you inspect or run it, then save the remaining uncertainty and decision.

Next, determine how these verification habits fit your course’s academic integrity and AI-use rules.