The log of questions your agent could not answer is the most useful thing it produces, and almost nobody reads it. Every entry is a runbook somebody already needed, written in the words they used, with evidence of demand attached. It is a documentation backlog that prioritised itself.
TL;DR
- A decline is not a failure. It is a free, dated, demand-ranked list of the documentation you are missing.
- Sort every entry into three piles: the content exists and is not written down, it exists under a word nobody uses, or the process genuinely does not exist.
- The vocabulary pile is the cheapest fix in all of IT support. No writing, just adding the words users typed to articles that should have matched.
- Track the rate rather than the count, or a quiet week reads as progress.
- The third pile is the valuable one, because it surfaces processes nobody ever decided rather than failed to document.
- If the log is empty, that is a configuration problem. An agent that never declines is guessing instead.
Why an empty log is bad news
Start here, because it changes how the rest reads. An agent reporting almost no unanswered questions is not covering your estate well. It is answering questions it should decline, which means confident wrong answers are going out and nothing is recording them.
A decline is a logged event and a wrong answer is not. That asymmetry is the whole reason to prefer an agent that refuses: the failure becomes visible and actionable rather than silent and acted upon. A deployment tuned to maximise answer rate is tuned to hide its own gaps.
So the first check on any new deployment is whether the decline log exists, whether anybody can export it, and whether it captures the question as typed. If it only captures a category, it is close to useless, because the exact wording is the part that tells you what to write.
The three piles
| Pile | What it looks like | Cost to fix |
| Documentation gap | Somebody on the team knows, nothing is written | An hour per topic, plus finding the right person |
| Vocabulary gap | The article exists under a term nobody uses | Minutes. Add the words, no writing |
| Missing process | Nobody has ever decided what happens | A decision, not a document |
Sorting takes about twenty minutes a month on a log of a few hundred entries, because you are grouping recurring themes rather than reading every line. Five or six themes will account for most of it.
The vocabulary pile is where the first pass should go, every time, because the return is absurd. Somebody asked about a dropped call and the article says session disconnect. Somebody asked about the staff portal and the article names the vendor product. Adding the user’s term to the existing article takes a minute and fixes the question permanently.
The third pile is the real finding
Some questions cannot be answered because nobody has ever decided the answer. What happens if I break a laptop while travelling. Who pays for a monitor at home. Can I install this myself. Who approves an exception to the device standard.
These are not documentation failures. They are undecided policy, and the agent has surfaced them by being asked repeatedly and having nothing to say. That is a genuinely valuable output and it is the one most likely to be mistaken for a product shortcoming.
Take this pile to whoever owns the policy as a list with frequencies attached, not as a conversation. Four people asked this in six weeks is an argument. We should probably clarify the device policy is not, and it is why these items sit undecided for years.
Running it monthly
Checklist:
- Export the declines for the month, with the question as typed and nothing identifying the person who asked.
- Group into recurring themes rather than reading line by line, and stop at the top six.
- Sort those six into the three piles, and act on the vocabulary pile the same day.
- Write one document per documentation gap, with the question as its heading and the answer in the first sentence.
- Send the missing-process pile onward with counts and dates attached, to a named person rather than a group.
- Record the unanswered rate, not the count, and compare it to last month rather than to a target.
- Re-ask the fixed questions next month to confirm they now answer, because a written document is not a working answer.
The last item catches the thing nobody expects. A new document can retrieve poorly, or can compete with an older article that says something different, so the question stays broken while the gap log shows it as handled. Verifying on the answer rather than the document is the only check that settles it.
Strip the identifiers first
One practical caution. The decline log is the most revealing dataset the system holds, because it is a list of things people did not know, attached to names if you let it be.
Nothing in the monthly routine above needs to know who asked. Export without identifiers, and if the product only exports with them, strip the column before anybody reads the file. A log reviewed by several people a month is a log that will eventually be read by somebody who should not have it, and the review is just as useful anonymous.
Disclosure: Matram is owned by the same people who publish PeopleOpsHQ. It is relevant to this topic because its strict mode declines rather than guessing, which is what turns a gap into a log entry instead of a confident wrong answer, and it is priced flat at $29, $69 and $199 a month with unlimited seats. It has no free tier, it answers questions rather than actioning tickets, and it is not an ITSM platform, so a declined question needs a named route behind it. The options are compared in our IT support chatbot comparison.
Final thoughts
Most teams read the decline log as evidence the agent is underperforming. It is the opposite: a dated, demand-ranked list of exactly which runbooks are missing, generated for free by people who needed them.
Twenty minutes a month, three piles, and the vocabulary pile cleared the same day. Then send the undecided-policy pile onward with counts attached, because that is the part no amount of writing fixes and the part this exercise is uniquely good at finding.
Frequently asked questions
What should you do with questions an AI agent could not answer?
Sort the recurring themes into three piles: content that exists in somebody’s head and is not written down, content that exists under a term nobody searches for, and a process nobody has ever decided. The first takes about an hour per topic, the second takes minutes and needs no writing at all, and the third is a decision rather than a document, which makes it the most valuable of the three.
Is an empty unanswered log a good sign?
No, it usually means the agent is answering questions it should decline, so confident wrong answers are going out with nothing recording them. A decline is a logged event and a wrong answer is not, which is the whole argument for preferring an agent that refuses, and a deployment tuned to maximise its answer rate is effectively tuned to hide its own gaps.
What is the cheapest fix in the log?
The vocabulary pile, where the article already exists under a term nobody uses. Somebody asks about a dropped call and the article says session disconnect, or asks about the staff portal and the article names the vendor product, and adding the user’s own words to the existing article takes about a minute and fixes that question permanently with no writing involved.
How do you raise the undecided-policy items?
As a list with frequencies and dates attached, sent to a named person rather than discussed with a group. Four people asked this in six weeks is an argument that gets a decision, whereas a general suggestion that the device policy could use clarifying is why these items sit unresolved for years despite everybody agreeing they should be sorted out.
Should you track the count or the rate?
The rate, because a quiet week with lower volume otherwise reads as an improvement when nothing has changed. Compare the rate to last month rather than to a target, since a target invites the behaviour that makes the log useless, namely configuring the agent to answer more and decline less regardless of whether it knows the answer.
Why re-ask the questions you have already fixed?
Because a written document is not a working answer. A new article can retrieve poorly, or can compete with an older one that says something different, so the question stays broken while your log records it as handled. Re-asking next month verifies on the answer rather than on the document, which is the only check that distinguishes work done from work that landed.
