Feedback & Learning

How votes, hand-over to staff, and learning from closed tickets work, every setting for them, and how to review what the bot learned

This page covers three things: the vote buttons under each answer, how the bot hands a ticket to staff when it cannot help, and how it learns new answers from closed tickets. It explains each setting, what staff and customers see in Discord, and how to review or undo what was learned.

Votes and learning are on the AI assistant → Atlas → Learning card (on the Help content tab). The hand-over settings are on AI assistant → Atlas → Answer questions in tickets (on the What Atlas does tab). Votes and learning are on for a new knowledge base, and hand-over is always on. Review learned entries under Staff Q&A and vote results on the Stats tab.

Go to AI assistant → Atlas → Learning

Feedback: what users see

Every answer the bot posts ends with 👍 and 👎 buttons when Let members rate answers is on. The same switch covers answers in tickets and in public help channels.

  • 👍 records a positive vote and replies Thank you! Your positive feedback helps improve our knowledge base. Pressing the same button again removes the vote.
  • 👎 opens a menu, What was the issue with this suggestion?, with the reasons Incorrect Information, Partially Correct, Outdated Information, Not Relevant, and Other Issue.
  • Picking any reason opens the Provide Detailed Feedback form with What could be improved?. Staff also get Correct Answer (helps AI learn); customers never see it. A submitted correction is confirmed with Correction Saved for Review and becomes a Waiting entry under Staff Q&A, marked From a correction. The bot uses it once someone approves it. See Corrections from the 👎 button.
  • In ModMail the buttons are on the answer in the member's DM, and only that member can vote. They are left off an answer long enough to be sent as several messages, and while Only count votes from staff is on.
  • With Only count votes from staff on, a member who is not staff gets Only staff members can vote on suggestions (error code TC002) when they press a vote button in a ticket.

Under every answer the bot also prints Generated by AI. Staff can verify important details. This note is always on and cannot be turned off.

Feedback settings

SettingChoicesDefaultWhat it does
Let members rate answers (on the Learning card)On, OffOnShows 👍 and 👎 under answers in tickets and help channel threads
Only count votes from staffOn, OffOffShown while Let members rate answers is on. Only staff can vote on answers in tickets; votes from other members are refused

In public help channels, who can vote follows Let members vote on the Answer in public help channels card instead: when it is on, every member can vote there; when it is off, only staff can.

Atlas uses the votes by itself: once a piece of content has a few votes, content members find helpful is used more often in later answers, and content they find unhelpful less. There is nothing to tune. Turn the switch off if you would rather not show vote buttons. Buttons already under earlier answers stay in Discord but stop counting votes: pressing one then shows Voting is turned off for these answers. (error code TC018). The same happens after the knowledge base or Atlas is turned off.

Escalation settings

Hand-over to staff is always on: when the bot has nothing to answer from, it offers the customer a Talk to Staff button instead of guessing, and it hands the ticket to staff when its own answer says a person needs to take over. These two settings are on AI assistant → Atlas → Answer questions in tickets:

SettingChoicesDefaultWhat it does
When Atlas can't helpStay out for the rest of the ticket, Stay out 1 day, Stay out 3 days, Stay out 7 days, Stay out 14 days, Stay out 30 daysStay out 7 daysHow long the bot stays out of a ticket after a hand-over, or after a missed response target escalates it. It comes back when the time passes, whether or not staff have replied; to keep it quiet once staff reply, turn on Stay quiet once staff reply on the same card. Stay out for the rest of the ticket means it never comes back in that ticket
Ping staff roles on hand-overOn, OffOffMentions your staff roles on the hand-over card. If the ticket's button has its own ping role, only that role is mentioned; with no ping configuration, all global staff roles are; with this setting off, nobody is pinged

What happens on escalation

  1. The customer asks something the bot cannot answer from your content, asks for something only your team can do (an account change, or a topic under Topics to leave to staff), or asks for a human (Can I talk to a real person?, human please, and similar in any phrasing).
  2. For a missing answer, the bot posts I don't have enough knowledge base information to answer that. Use the button below to request staff help. with a 🙋 Talk to Staff button. The ticket is not handed over yet: the bot keeps answering the customer's next questions. After a button is posted, further unanswered questions in the next 5 minutes do not post another one. The button works for the ticket creator and staff only.
  3. In ModMail the same message and button arrive in the customer's DM, where only they can press it, and a 🙋 Talk to Staff Offered card in the ticket channel shows staff the question. Nobody is pinged by that card.
  4. When the customer presses Talk to Staff, a Staff Assistance Requested card is posted in the ticket (or Escalated by Staff if a staff member pressed it) and they see A staff member has been notified and will be with you shortly! A second press replies This ticket has already been escalated to staff.
  5. When the customer clearly asks for a person, the ticket is handed over straight away and they are told I've shared our conversation with the team. A team member will be with you shortly., in ModMail too. Staff get a Human Assistance Requested card in the ticket. It includes a short Summary of the issue and what was already tried, so staff can pick it up without reading back. If no summary can be written, the card is posted without it.
  6. When the bot's own answer says staff should take over, it posts a 🤖 AI Escalation card in the ticket showing the customer's question under User Question. When several messages arrived together the title reads 🤖 AI Escalation (Batched).
  7. After a hand-over (steps 4 to 6), the bot stays silent in that ticket for the time set in When Atlas can't help.

Learning

Learning turns the answers your staff give into knowledge base entries. There is no command to run. Choose how it works on AI assistant → Atlas → Learning, under New answers from solved tickets:

ChoiceWhat happens
OffAtlas does not learn from solved tickets
Ask me firstEach new answer waits in Staff Q&A until you approve it. Suggested updates to existing entries wait too
Add automatically (default for a new knowledge base)New answers Atlas is sure about are added right away and posted in the alerts channel; the rest wait for review

The same choice covers solved threads in public help channels. Learning needs Answer questions in tickets on.

The Learning card has these settings too:

SettingChoicesDefaultWhat it does
Post each new answer in the alerts channelOn, OffOnShown while Learning is Add automatically. Posts an Auto-Learned card with an Undo button for each answer added automatically. When off, nothing is posted for them; review and delete them under Staff Q&A instead
Tidy up answers regularlyOn, OffOnCombines duplicate learned answers into one and archives old learned answers nobody uses. Runs only while Learning is not Off
How oftenWeekly, Every 2 weeks, MonthlyMonthlyShown while Tidy up answers regularly is on

What happens when a ticket closes

  1. The ticket is closed by staff, by inactivity auto-close, or because the user left the server. Tickets with fewer than 3 messages are skipped, and so are tickets from members on Keep Atlas away from or from a panel button with Mute AI replies on.

  2. The bot extracts question-and-answer pairs where a staff member gave the answer. Personal data is redacted, and channel links are only kept when they point to a channel in your server.

  3. Each pair gets a confidence score.

  4. Add automatically, and Atlas is sure about the pair: the pair goes live and is searchable immediately. While Post each new answer in the alerts channel is on, an Auto-Learned from Ticket #123 card is posted to the AI alerts channel with the confidence, category, Question, Answer, and an Undo button. Undo removes the entry from search and marks it reverted. At most 50 pairs a day go live this way.

  5. Every pair with Ask me first, and pairs Atlas is less sure about, that may still contain personal data the redaction missed, or over the daily cap: the pair is saved and waits for review. A Pending Review card for the ticket (for example Ticket #123) with ✅ Approve, ✏️ Edit, and 🚫 Reject buttons is posted to the AI alerts channel, and the pair appears under Staff Q&A as Waiting.

  6. Overlapping an existing entry: a duplicate is dropped, an enrichment adds a detail to the existing answer, and a supersession replaces the outdated fact.

    With Add automatically, updates to entries the system learned itself and nobody edited are applied automatically (unless the new pair may still contain personal data) and post a KB Entry Updated from Ticket #123 or KB Entry Superseded from Ticket #123 card with the previous and updated answer and an Undo button. These cards are posted even while Post each new answer in the alerts channel is off, because their Undo button is the only way to reverse the update. Other updates, and every update with Ask me first, appear under Staff Q&A marked Suggested update, showing the Suggested new answer or Suggested extra detail; approving applies the change to the existing entry instead of creating a new one. With Add automatically, a suggested update learned from a closed ticket that meets the same rules as an automatic one (for example, one held back because 20 updates were already applied that day) is applied by itself once it has waited 7 days without a decision, and posts the same card with an Undo button. Approve or ignore it before then to decide yourself.

  7. While Tidy up answers regularly is on (the default), Atlas tidies up the answers it learned as often as How often says (monthly by default): it merges duplicates into one answer and archives old unused ones, and a Knowledge Base Consolidated card reports how many were merged and archived.

The AI alerts channel is set under General → Where the bot posts; without one, cards go to your logs channel.

Reviewing on the Staff Q&A tab

Go to AI assistant → Atlas → Staff Q&A. The Staff Q&A tab shows the list, and a count on the tab says how many answers are waiting for review.

  • The status filters show how many entries each status has: All, Waiting (waiting for review), Approved, Approved by Atlas (went live because the bot was sure enough, with how sure it was), and Ignored. On phones they are a dropdown. Search looks through questions and answers.
  • Each waiting entry has Approve and Ignore buttons. The ⋯ button on an entry holds Edit and approve (for waiting entries) or Edit (for approved ones) and Delete. Entries changed by a merge show Updated once or Updated 2 times style badges; edited entries show Edited; entries the bot uses show In use.
  • Add an answer adds a hand-written question and answer that is available to the bot immediately.
  • Entries created from an uploaded file show the file's name. A waiting one lists the links, email addresses, or numbers in its answer that the file does not contain. Entries from a 👎 correction show From a correction and who sent it.
  • The ⋯ button next to Add an answer holds Refresh and Delete all waiting answers, which deletes every entry still waiting for review. Approved and ignored entries are untouched. This cannot be undone.
  • Deleting an approved entry also removes it from search. If an entry was changed elsewhere while you were editing, the dashboard reloads the latest version and asks you to try again.
  • The assistant above the list can filter the list for you and suggest entries to delete. Nothing is deleted until you confirm in the dialog, where you can untick entries to keep.
  • Every way entries are created, and how the bot uses them, is on Q&A Entries.

Viewing feedback data

Open the Stats tab on AI assistant and choose Votes by source:

  • A time range at the top: 7 days, 30 days, 90 days, or All time.
  • Answers given, Votes (helpful and not helpful), Found helpful (the share of 👍 votes), and Sources with votes, with a count of sources that need a look because few members found them helpful.
  • Each source: every source with how often its answers were found helpful and its 👍 and 👎 counts, marked Needs a look when that share is low. Search it by name and sort by Most votes, Most helpful first, or Least helpful first.
  • Parts of your content: the individual pieces of content Helping the most and Worth a look.
  • Why members voted thumbs down: the share of 👎 votes by reason, so you know whether content is wrong, outdated, or off-topic.

Blocks without enough votes yet are listed together under Not enough data yet. Refresh is in the ⋯ menu next to the time range.

Questions answered on the same tab shows how the bot handled questions over 7 days, 30 days, or 90 days:

  • Questions, Answered, Handed to staff, and Checked against your content.
  • A Questions per day chart.
  • What happened to each question (answered, not enough info in your content, asked the member to clarify, handed to staff, and rarer outcomes when they happened).

More details below it shows how often the bot reworded a question to search better, shortened long content, and passed or failed its fact check, why tickets went to staff, and the votes on these answers.

Limits and gotchas

Learning cards and review cards need an AI alerts channel or a logs channel. Without either, learning still happens and pending entries still appear under Staff Q&A, but nothing is posted to Discord and there is no Undo button; delete the entry in Staff Q&A instead. Suggested updates then always wait for you.

  • Automatic updates never touch an entry a human created or edited. Those always become review proposals.
  • Delete all waiting answers and Delete are permanent.
  • Only staff can submit a correct answer through the 👎 form. It always waits for approval under Staff Q&A and never posts a Pending Review card.

Troubleshooting

SymptomCauseFix
No vote buttons under answersLet members rate answers is offTurn it on; answers posted before the change keep their old layout
TC018 when votingVoting is unavailable for that messageSee TC018
The bot keeps answering after a hand-overThe time in When Atlas can't help passed. It comes back then even if staff have repliedChoose a longer time or Stay out for the rest of the ticket, or turn on Stay quiet once staff reply
Nobody is pinged on hand-overPing staff roles on hand-over is off, or the ticket's button has a ping role set to noneTurn it on and check the button's ping configuration
No Auto-Learned or Pending Review cardsNeither an AI alerts channel nor a logs channel is set, or the bot cannot post thereChoose a channel under General → Where the bot posts; waiting entries still appear under Staff Q&A
No Auto-Learned cards, but Pending Review cards arrivePost each new answer in the alerts channel is offTurn it on under AI assistant → Atlas → Learning, or review new answers under Staff Q&A
Members get TC002 when voting in a ticketOnly count votes from staff is onExpected. Turn it off to let every member vote
Every learned answer waits for reviewLearning is set to Ask me firstApprove them under Staff Q&A, or switch to Add automatically
Nothing is learned from a closed ticketLearning is Off, the ticket had fewer than 3 messages or no staff answer, or Atlas is kept away from that ticketCheck the Learning choice and Keep Atlas away from
Nothing is learned from any closed ticket, and the Learning card says Paused while AI usage is set to Essentials onlyThe bot owner set AI usage to Essentials only, which pauses learning from closed ticketsThe bot owner chooses Default on AI assistant → Atlas → AI usage and saves
The bot learned something wrongA staff answer in a closed ticket was wrongPress Undo on the card, or Edit or Delete the entry under AI assistant → Atlas → Staff Q&A
An approved entry was rewrittenAn automatic update replaced a system-learned entryPress Undo on the KB Entry Superseded card; edit the entry once so future updates become proposals

Common questions

Does the bot learn from every closed ticket? Only when Learning is not Off, AI usage is not set to Essentials only, the ticket has at least 3 messages, a staff member answered in it, and Atlas is not kept away from the ticket. With Ask me first every new answer waits for approval; with Add automatically confident answers go live, at most 50 a day, and the rest wait.

Where do I approve what the bot learned? Under AI assistant → Atlas → Staff Q&A, or with the Approve, Edit, and Reject buttons on the Pending Review card in your alerts channel.

How do I check every new answer before the bot uses it? Set Learning to Ask me first. Nothing learned goes live until someone approves it.

How do I undo an auto-learned answer? Press Undo on the Auto-Learned card in the alerts channel, or delete the entry under Staff Q&A. With Post each new answer in the alerts channel off there is no card, so use Staff Q&A.

Can I stop members from voting and keep votes for staff? Yes. Turn on Only count votes from staff under Let members rate answers on AI assistant → Atlas → Learning. In help channel threads, turn off Let members vote on the Answer in public help channels card instead.

How often does Atlas clean up learned answers? Monthly by default. Change it with How often under Tidy up answers regularly on the Learning card, or turn that switch off to keep every learned answer as it is.

Can customers teach the bot wrong answers through the 👎 form? No. Only staff get the Correct Answer (helps AI learn) field, and every correction waits for approval under Staff Q&A before the bot uses it. Votes only move existing content up or down.

How long does the bot stay quiet after handing over? The time set in When Atlas can't help: 1, 3, 7, 14, or 30 days, or the rest of the ticket. The default is 7 days. It comes back when the time passes, whether or not staff have replied. To keep it quiet once a staff member answers, turn on Stay quiet once staff reply on AI assistant → Atlas → Answer questions in tickets.

Can I turn off the hand-over to staff? No. Atlas always offers a way to reach staff when it cannot help, so a member is never stuck with a bot.

Can I turn off the "Generated by AI" note? No. The transparency notice is always shown under AI answers.

Last updated on

On this page