Questions we couldn't afford to answer

26 Sep 2026 · 7 min read · Draft

Most of the talk about models at work is about volume: more code, sooner. That’s real, but I don’t think it’s the most interesting change. The bigger one, for me, is what’s happened to the cost of an answer.

We always asked the questions. Where are users dropping out? Are we tracking to the estimate? Will this schema hold up? How is this person really doing? What we couldn’t do was afford the answers, and in an agency that constraint is sharper than most. Time is sold by the hour, so analysis is either billable, which means a client has to agree to pay for it and often won’t, or it’s overhead competing with utilisation. A specialist tool per client rarely justifies its licence, and the data sits in each client’s own accounts. So the questions got a quick look, an educated guess or a line in a retro, and we moved on.

That’s the part that has changed. Answers that used to cost days, a tool or an analyst now cost an hour or less. That’s cheap enough to fold into delivery without a line item, and cheap enough to do properly in the gaps between billable work.

Four answers that used to cost too much

What is the analytics telling us? Every app we build ships with analytics, and clients rightly ask what it says. A proper answer used to mean an analyst’s time the client hadn’t budgeted for, so it usually meant a skim of the dashboard. Give a model the raw event export and the question “where are people giving up, and what would you change first?”, and a first analysis with ranked recommendations takes minutes. That makes it cheap enough to include in a sprint review rather than sell as a separate piece of work.

How are we really doing against the estimate? Comparing estimates with progress properly meant a delivery analytics platform, which is hard to justify per client, or someone reconciling Jira exports with timesheets by hand, which is non-billable and always the first thing dropped. Now the tickets, worklogs and original estimate go straight into a model, and it can show where the time went, which kinds of work we consistently misjudge, and what that means for the rest of the project. There’s no platform and no integration project, and the answer is ready before the client conversation it’s needed for.

What should this database look like? Trying several schema designs against realistic data and real queries was always the right thing to do. On a fixed-price project it was rarely affordable, so we went with experience and a whiteboard. Now several designs can be tested properly in the time a discovery phase allows: generate realistic data, run the queries the product will need, and see how each design copes as it changes. The analytics can be designed in from the start, too. If we know which funnels and events the client wants to measure, the schema accounts for them rather than having them bolted on after launch.

What does someone’s recorded work actually show? This is the one that surprised me by how far it goes. I recently reviewed someone’s work where I had little day-to-day visibility. Their project data was all there, but reading months of it by hand would have taken days of non-billable time I didn’t have, so my assessment would have rested on a handful of conversations and whatever feedback happened to reach me. Instead, I had a model work through their pull request history. It looked at how they responded to review comments, whether a reviewer’s point led to changed code or to a discussion, what their tests actually exercised, and how a feature they’d built had grown in the source. A subjective judgement like “takes feedback seriously” could be tied to recorded behaviour, with examples, and the review was fuller and fairer for it.

What they have in common

None of these are about producing more. They’re about knowing more before deciding.

• The questions and the data were always there. Event logs, tickets, worklogs and commit history have existed for years. What changed is that reading them properly became affordable.
• The answer fits inside the work. At minutes or an hour, analysis no longer needs its own budget line, its own tool or a client’s sign-off. It becomes part of doing the job well.
• Impressions become claims someone can check. “The onboarding feels leaky”, “we always underestimate integrations” and “they take feedback well” all turn into statements with evidence attached, which someone can challenge.
• Breadth beats memory. A rushed analysis leans on the few moments that stuck. A model reads everything, so quiet good work, rare failure paths and slow drifts get counted too.
• The expensive part moves. Answering stops being the hard part. Asking a good question and checking the answer become the work, and in an agency those are the parts clients should be paying for.

Where it goes wrong

A model-written change goes through review and CI. A model-written analysis tends to be believed, especially when it arrives with numbers and a confident recommendation attached. The failure modes are different from code, and none of them show up on a coding benchmark:

• Misreading the data. An event that fires twice, a status that means something different on this project, an estimate field someone reused for something else. The analysis is internally consistent and built on a wrong assumption.
• Evidence that doesn’t exist. A quoted comment from the wrong pull request, a figure that can’t be traced back to a row, or a pattern described with more confidence than the data supports.
• Answering the question as asked. Ask where onboarding is failing and you’ll get an answer even if it isn’t. Ask for evidence of a weakness and you’ll get some. The model finds what the framing invites.
• Confident figures from a thin sample. A drop-off rate across a week of traffic, or a median across a dozen review threads, reads like a fact.
• Correlation served as recommendation. “Users who do X retain better, so push everyone to X” is the classic, and it’s easy to accept when it arrives as a tidy action.
• Agreeing with the person asking. A drift towards confirming what you already thought, which is the most dangerous when the analysis is about a person.

Method: the model as researcher, not judge

The pattern that works for me in all four cases is to treat the model as the researcher, and keep the judging for myself.

• Check the inputs first. Before trusting any analysis, have the model explain what each field and event means, and correct it where it’s wrong. I’d expect most bad analyses to fail here rather than in the reasoning.
• Trace every figure back. A number or quote that can’t be traced to a row, a ticket or a commit doesn’t go into anything I share.
• Ask for the other side. Ask for the counter-evidence, the alternative explanation and the case against the recommendation as explicitly as the finding itself.
• Keep the method visible. Whatever I write up says what data was used, over what period and how, so someone else can check it or challenge it.
• Re-run it. Ask the same question twice, or phrase it differently. If the answer moves a lot, it wasn’t much of an answer. That’s the variance point from my last post, applied to analysis.

When the data is about people

The first three examples analyse products and projects. The fourth analyses a person, and that deserves its own care. I haven’t written this section yet; these are the points it needs to cover.

• Transparency: the person should know what was analysed and how, and be able to challenge it.
• Surveillance creep: the same tooling could run continuously on everyone. Where’s the line?
• Measures become targets: publish “time to act on review comments” and people will optimise for it.
• Data protection: this is processing employee data, and it goes to a model vendor. Where it goes and who can see it matters.

Close

If models only made us faster at writing code, the change would be large but familiar. What I find more significant is that they’ve made answers cheap, and in a business that sells time, cheap answers change what good work looks like. The questions we used to wave through with a guess can now be answered as a matter of course. The value moves to asking the right one, spotting the misread field and refusing the confident number that doesn’t trace back to anything. That’s the same skill it’s always been, only now we can afford to use it every time.