How we measure what AI says

Honest data means the count and the range on every figure, statistical significance stated where it can be, and the answers themselves attached so the reading can be checked.

What makes our readings different

  • Fair. Every company in a category gets the same questions in neutral words, client or not, because a reading that favours the client is worth nothing to the client.
  • Honest. Every figure shows its count and its range, and we state significance where we can. A single run is labelled an observation until it has been repeated.
  • Traceable. Every incorrect fact is traced to the pages the model drew on, and where we can we name the content that influenced the answer.
  • Repeatable. The same questions are asked in the same words at every reading and versions are kept, so change is measured rather than guessed.
  • Counted, not judged. We do the calculations outside the AI wherever we can, and the tools we build are AI and platform agnostic. A share is counted from the answers, never estimated by a model. Where a model's judgement is used, such as reading which way an answer leans, the figure says so.
  • Careful. We say what moved and what else changed at the same time, and we do not claim a cause. A company that is not mentioned is never given a score of zero.

Our AI survey framework

We ask the AI models the same questions a customer survey asks people, and then we compare the two sets of answers.

1. Start with a survey that already exists

Yours or a public one. For UK banks we use the CMA service quality survey, which asks customers whether they would recommend their bank, its app, its overdraft and its branches.

2. Ask the AI models the survey question about each company in the category

"Would you recommend Northbank as a main current account? Yes or no, and the main reason." Every company in the category is asked in the same words.

3. Ask each question several times and count the answers that say yes

Five models, three repeats of every question, with web search off and on. The AI score is the share of answers that say yes, and the survey score is the share of customers who would recommend. The two sit on one scale.

4. Add the brand measures that matter to you

Trust, value, innovation or the claims you want checked. The same rule applies to each one: every company is asked in the same words.

What the measures mean

AI visibility

The share of AI answers that name the company when asked about its category, counted across models and repeated runs.

Named and picked

Of the answers that name the company, the share that recommend it over its rivals.

Head-to-head preference

Of the answers to a comparison question, the share that favour the company, the share that hedge, and the share that favour the rival. For the hedged answers, which company was preferred for which need, and the words the models used for it.

Cited but not named

Of the answers that cite one of the company's own pages, the share that do not name the company. For those answers, who was named instead and what for.

Share of voice in AI

The company's share of all brand mentions across the answers in its category. The same idea as share of voice in media, counted in AI answers.

Brand perception

How the models and how people score the company on the same measures, such as trust and value. The gap between the two is the finding.

Fair questions, measured on Model Research IQ

We write down what we expect before each test, and we publish the result whether it holds or is overturned.

Questions about the method are welcome. Our details are on the contact page, and the platform is described on its own page.