Training and Tuning Your AI Agent for Better Answers
Improve your AI agent's accuracy by refining KB content, adjusting scope, configuring fallback behavior, and using the dashboard to close answer gaps.
Getting your AI agent connected is the first step — getting it to consistently deliver accurate, helpful answers takes ongoing refinement. Even when your KB articles are well-written, the agent benefits from deliberate tuning: directing it toward the right content, telling it what to do when it's stuck, and regularly updating articles based on the questions customers actually ask. This page walks you through each lever you control and a repeatable workflow for closing the gaps your agent reveals.
The Three Tuning Levers
You have three primary controls over your agent's answer quality. Adjusting all three in combination produces the most significant improvements.
1. Content Quality
The agent can only be as good as the articles it has access to. Thin, vague, or outdated articles produce thin, vague, or outdated answers. To improve content quality:
- Write for specific questions. Each article should address a distinct topic or task. An article titled "Getting Started" that covers ten different things is harder for the agent to match accurately than ten focused articles.
- Use the language your customers use. If customers ask "why won't my password reset?", your article should use that phrasing somewhere — not just the technical term "credential recovery flow."
- Keep articles current. An article describing a UI that no longer exists will produce an incorrect answer. Review and update articles whenever your product changes.
- Complete every article. An article stub with placeholder content ("details coming soon") teaches the agent nothing. Publish articles only when they're complete.
2. Scope Configuration
Your agent's scope determines which products and topic categories it draws from when answering questions. A broadly scoped agent pulls from your entire KB, which can cause it to surface articles from unrelated products when a more specific match exists.
To tighten scope:
- Open External KB → AI Agents, select your connection, and click Edit Scope.
- Enable only the products and categories relevant to the surface where the agent is deployed.
- If you run multiple agent deployments (for example, one agent embedded in your app and another on your website), give each its own connection with its own scope.
3. Fallback Behavior
Fallback behavior controls what the agent says when it can't find a matching article with sufficient confidence. The default fallback message is:
"I don't have specific information on that topic. To get help from our support team, please [contact support link]."
You can customize this message in your AI agent platform's response settings. A well-configured fallback:
- Acknowledges that the agent doesn't have the answer (rather than producing a low-confidence guess).
- Gives the customer a clear next step — a link to your support contact form, a phone number, or a prompt to open a ticket.
- Optionally logs the unanswered query so it appears in your low-confidence dashboard for review.
Reviewing Agent Performance
Your AI agent dashboard tracks every query and assigns each a confidence score based on how well the retrieved articles matched the question. Queries that fall below the confidence threshold appear in the Low-Confidence Queries report.
This report is the most actionable signal you have for improving your KB. Each entry tells you:
- The exact question the customer asked.
- Whether any article was returned (low-confidence match) or no article was found (no match).
- The article that was returned, if any, so you can evaluate whether it was actually relevant.
Review this report at least once a week when your integration is new, and monthly once it stabilizes.
Tuning Workflow
Run through this cycle whenever you want to improve your agent's answer quality or after any significant product change.
In your KB dashboard, go to External KB → AI Agents → [Your Connection] → Performance. Filter the report to the last 7 or 30 days and sort by query volume so the most common gaps appear first.
For each low-confidence query, determine the cause:
- No matching article exists — You need to create a new article covering that topic.
- A relevant article exists but wasn't matched — The article may be too broad, use different terminology than the customer, or be filed under a category not in the agent's scope. Update the article or adjust the scope.
- The article is outdated — The article exists and matches, but its content no longer reflects your current product. Update the article.
Group related queries together — five variations of "how do I cancel my subscription?" all point to the same missing article.
Write new articles or revise existing ones to address the identified gaps. Follow the content quality guidelines above: be specific, use customer-facing language, and complete the article fully before publishing. Publish each article when it's ready — unpublished drafts are not indexed by the agent.
Once your articles are published, trigger a manual re-sync so the agent picks up the new content immediately rather than waiting for the next scheduled sync cycle. Go to External KB → AI Agents → [Your Connection] and click Sync Now. The sync status indicator shows when the process completes.
Use the Test Query tool on your connection page to replay the specific questions that previously produced low-confidence results. Verify that the updated articles are now being retrieved and that the response content accurately addresses the question. If a query still returns a low-confidence result, revisit the article and refine the language further.
Your support ticket backlog is the best source of article ideas. Export the last 90 days of tickets, identify the questions that came up most frequently, and use each one as the title of a new KB article. Writing articles around real customer questions — in the exact phrasing customers use — dramatically improves how accurately the agent matches incoming queries.
By default, your agent platform re-indexes your KB feed every 24 hours, automatically picking up any articles you've published or updated since the last sync. If you need the agent to reflect a change immediately — for example, after correcting an inaccurate article — use the Sync Now button on your connection page to trigger an on-demand re-index. On-demand syncs typically complete within two to five minutes depending on the size of your KB.
Frequently Asked Questions
The most direct way to prevent the agent from answering on a topic is to remove that topic's articles from the agent's scope. Go to External KB → AI Agents → [Your Connection] → Edit Scope and deselect the product or category containing the relevant articles. The agent will no longer retrieve those articles when responding to questions.
If you want to keep the articles accessible to customers browsing your KB but hide them from the agent specifically, you can create a dedicated topic category for agent-excluded content, file those articles there, and remove that category from the agent's scope.
Note that simply unpublishing an article removes it from the feed on the next sync, which also prevents the agent from using it — but unpublishing also removes it from your public-facing KB, so customers browsing manually won't find it either.
Yes. Every connection has a built-in Test Query tool accessible from the connection's detail page under External KB → AI Agents → [Your Connection] → Test. Enter any question to see exactly what the agent retrieves from your feed, including the matched articles and their relevance scores. The test runs against your live published content, so you can publish an article and immediately test whether the agent picks it up before enabling the connection for customers.
If your AI agent platform also has a staging or sandbox environment, you can point a separate KB connection at it with the same credentials and test the full end-to-end conversation experience without affecting your production agent.
A manual Sync Now re-index typically completes in two to five minutes for a KB with up to a few hundred articles. Larger KBs with thousands of articles may take up to fifteen minutes. You can monitor progress in real time on the connection detail page — the sync status updates as each product's articles are processed.
Scheduled automatic syncs run in the background every 24 hours and follow the same timing. If a scheduled sync is already in progress when you trigger a manual sync, the manual sync is queued and begins immediately after the scheduled sync completes.