I gave an API a filter called total_banana_pancakes. Same data.

A 200 with plausible rows is not proof a filter applied. The nonsense-value control that proved it, a sort that ranked 1 click above 257,561, and my rule now.

At my day job I was checking whether a published prompt for a reporting API did what it claimed: exclude one kind of record from a report. So I ran the call three ways. No filter. The real filter. And a filter called total_banana_pancakes, a value I invented on the spot because I was in a hurry and hungry.

All three came back byte for byte identical. 183 rows, 795,585 clicks, the same hash on the response body.

What it looked like from the outside

The prompt had been published with a sentence saying the exclusion was applied. The call returned a 200. The rows were sensible: the right columns, numbers in the right ranges, a total that matched the dashboard. Anyone reading the output would have said it worked, and until that afternoon, anyone had. Including me, the week before, when I wrote the sentence.

A response that looks right is very persuasive. It is the same persuasion as a confident chatbot answer: fluent, complete, and not evidence of anything in particular.

What was actually true

The API accepts whatever you send, ignores what it does not understand, and returns a cheerful 200 with plausible numbers. It never validated the filter’s name. To prove the mechanism itself was fine, I ran a fourth call with an exclusion I knew was real, removing one large partner’s traffic. That dropped the result to 182 rows and 30,765 clicks. So exclusions work. The API simply has no opinion about whether the one you typed exists.

Which leaves an uncomfortable question I still cannot answer: was the real filter in the published prompt wrong, or were there simply no rows of that kind to exclude that week? The API would have returned the same thing either way. Nothing was wrong with the data. Something was wrong with my idea of evidence. Sensible-looking rows are not proof the filter applied. They are proof the server was switched on.

The same API, a month earlier, had taught me the other half of this and I had not generalised. Its sort parameter accepts a metric name that does not exist, echoes the fake name back in the response as if it were valid, and returns the rows in whatever order they came. I checked: the item with 257,561 clicks came back second, behind one with 1 click. The documentation, when I finally read it properly, said so in six words: “unknown names are silently ignored.” So “top ten partners by revenue” quietly becomes “some ten partners”, and the output looks exactly like a ranking because it has ten rows and a heading.

The control that costs one extra call

The fix is not clever. Every claim that a filter, an exclusion or a sort did something now gets a control with a made-up value. If the nonsense version returns the same thing as the real version, the real version has proved nothing, and I go and find out why. That is one extra API call per claim, and it has already paid for itself more than once.

It needs a second half, though, or it is just a different way of fooling yourself. A negative control (nonsense must do nothing) tells you the parameter is being ignored. A positive control (a value that must change the result) tells you the mechanism works at all. The partner exclusion was the positive control. Without it I could have concluded “exclusions are broken” when in fact “exclusion names are unchecked”, which is a different bug with a different fix.

The rule I follow now: before I believe a filter, a sort or an exclusion did anything, I send one value that must do nothing and one that must do something, and I compare all three responses by hash, not by eye. If I cannot think of a value that must change the result, I do not yet understand the call well enough to publish a claim about it.

Why this is going into the visibility checks

My side project asks five AI assistants the same buyer questions three times each and counts how often a product is named. The dangerous output there is not “no”. It is “yes”. “The AI named you” is exactly the kind of answer that looks right whether or not it is. A text match finds your name in an answer that quotes your own page on the way to recommending two competitors. A small model asked “was this product recommended?” says yes to a cameo.

So the counting pipeline gets the same two controls. A planted product name that does not exist, which must come back as “not named” every time, or the yes machine is saying yes to nonsense. And a question with a well-known product in the category, which must come back as “named”, or the machine cannot find a name even when there is one. The controls run with every batch, and a person reads the disagreements. The person is me. I like banana pancakes, and I no longer trust a 200.

See if AI has heard of you

Five real buyer questions, five AIs, seventy-five answers, and a person reads your report. Results by email in two working days. Free.