What confidence should mean in AI-assisted research
In research, confidence should describe the evidence behind a finding: how many independent sources support it, whether any contradict it, and how recent they are. It should not describe how fluent or certain the generated text sounds. A model can write a certain-sounding paragraph on thin evidence.
Three things confidence should reflect
- Support: how many independent sources back the claim, and whether they are the right kind of source for it.
- Conflict: whether any source contradicts it, and how weighty those are.
- Directness: whether the source states the fact or the claim is inferred from it.
Freshness is a separate property. A well-supported claim can be a year old. Show it beside confidence, not folded into it.
A rubric a person can apply
| Label | What it means | Example (synthetic) |
|---|---|---|
| High | Stated directly by an authoritative source and confirmed by a second, with no contradiction | Meridian lists a $59 Team plan on its pricing page and in its announcement |
| Medium | Directly stated by one source, or supported by several indirect ones, with no strong contradiction | Three reviewers describe slow setup; the vendor does not address it |
| Low | A single indirect source, or sources that conflict | One anonymous forum post says Meridian dropped its free plan |
What confidence is not
- A model's own stated certainty. Models can sound sure about unsupported statements.
- A percentage with false precision. “83 percent” implies a calibration nobody did.
- A measure of importance. A low-confidence claim can still matter a great deal, and a high-confidence one can be trivial.
Importance and confidence together
Show both. A high-importance, low-confidence item is a prompt to verify. A high-importance, high-confidence item is a prompt to act. A low-importance, high-confidence item can wait. Mixing them into one score loses that distinction.
A worked example
Fieldnote's analyst sees two items on the same morning. “Meridian raised the Team plan to $59” comes from a before and after on the pricing page: high confidence, high importance. “Meridian may be planning an agency tier” comes from one job posting: low confidence, medium importance. The first goes to the pricing review. The second goes on a watchlist, with a note on what would raise its confidence, such as an announcement or a docs page.
What the reader needs to see
- The label, and the sources it rests on, one click away.
- Whether any evidence contradicts it.
- How old the newest source is.
- What would change the label.
How to calibrate the labels over time
Keep a short record of items labelled high that later turned out wrong, and items labelled low that turned out right. Review it each quarter. If high-confidence items are failing, the bar for high is too low, and the rubric needs a stricter definition of independent sources.
Confidence in Sigblaze
Sigblaze shows confidence as a low, medium or high label alongside importance and freshness. Freshness is computed from source dates, not from a model's opinion. The labels are a guide for review, and a person still approves what is reused.
Questions
Can confidence be a number?
It can be stored as one, but a label is easier to review and does not claim precision that was never measured.
This piece contains no external statistics. Fieldnote and Meridian are fictional companies, and the figures in the examples are synthetic.