Sentiment Checks

Sentiment Checks answer one question about a period from what your community said on Discord and Steam. You ask in plain words. BetaHub works out what to measure, counts the people who talked about it, and writes a short report with real quotes.

Three kinds of check

You don’t have to pick the kind: BetaHub works it out from your question. A fix check also covers balance changes (a nerf or buff) and performance fixes.

When to use

• You shipped a fix and want to know whether players stopped hitting the problem.
• You want to compare how two releases were received, without a bigger or louder community skewing the result.
• You ran a sale, free weekend or creator campaign and want to know how your regulars took it and what new players talked about.

Requirements

• Pro or Enterprise plan, the same plans as Sentiment Analysis. On other plans the page explains this and you can’t start a check.
• A role that can view analytics.
• Sentiment Analysis collecting messages from your Discord channels or Steam Discussions for the period you ask about. A check only sees messages collected there.

Asking a question

1. Open Sentiments and click Sentiment checks.
2. Type your question and click Ask. To pick the kind first, click its button on the cards below, such as Start a fix check, then type your question and click Ask.
3. Answer a few questions, if asked. BetaHub first looks at your releases, bug reports and messages. When something is unclear, such as the fix date or which platforms got it, it asks up to four questions at a time, often with an answer suggested from your data. Pick an option, type your own under Something else, or add a note, then click Send answers. Not sure is a fine answer; anything BetaHub had to assume is listed in the plan under Assumed.
4. Confirm the plan. BetaHub shows what it will analyse: for example the problem and the words players use for it, the fix date per platform, the periods compared, and the staff roles it leaves out under Left out (staff). Click Run check, or type what should change (for example “the fix was on Sep 20”) and click Change plan.
5. Wait for the report. A check usually takes a few minutes. The page updates by itself for up to 20 minutes (reload it after that), and you can leave it: the check keeps running and you can reopen it from Your checks. If the check is interrupted, for example by a BetaHub update, it continues where it stopped.

Your checks lists the project’s 25 most recent checks with their state, such as Needs your answer or Ready to run, and its verdict once it’s done. You can stop an open check with Cancel check.

If community messages can’t answer the question at all, such as sales numbers or player counts, BetaHub explains why and the check ends with Can’t answer from chat.

Reading the report

Every report leads with the answer, then shows what it rests on:

• Verdict and headline: the answer in a few words and one sentence, with the evidence level (Strong, Moderate or Weak evidence) and the date it was saved.
• Main chart: the numbers behind the verdict. It differs per kind; see the next sections.
• What players describe: a short summary and the themes found, each with a message as proof.
• Quotes: real messages next to themes, side effects, patch-note items and new problems, with their channel, date and the author’s roles. Discord messages have an Open in Discord link.
• Worth knowing: caveats that limit the result, such as a quieter chat, many new players, or only a few days of data.
• How this was worked out and How this check was set up: the method, and your conversation with BetaHub before the check ran.

In a fix check

• Main chart: people complaining in the days before the fix next to people complaining since, with the change per active person, so a busier or quieter chat doesn’t skew it. Platforms that got the fix on different days are shown separately, each measured from its own date.
• Straight from chat, since the fix: quotes from people who say it’s Still broken and people who say it’s Fixed or better, with how many people said each. For a balance change the two sides are Like the change and Miss the old version.
• Possible side effects of the fix: problems at least 3 people raised since the fix that nobody (or only one person) raised before.
• The people who complained before: how many are still around and stopped mentioning it, still report it, or haven’t posted since. Discord only.
• Complaints per day, per 1,000 active people, with the fix date marked.

For a balance change, the main chart shows how people reacted: liked it or found it fair, miss the old version, or say it went too far or not far enough. From 10 days after the change, it splits the first week from the days after, so you can see whether a backlash is fading.

In a release report

• Main chart: each release’s mood against its own usual (the 14 days before it came out), for everyone and for returning players. Comparing each release with its own usual means a bigger or louder community doesn’t win by volume.
• How each patch-note item landed: a result per item, with how many people liked or disliked it. Items come from the release’s summary and description in BetaHub; without patch notes, this part is empty.
• New problems not in the patch notes.
• Mood by day since each release.

In a campaign recap

• Main chart: your regulars’ mood during and after the campaign against their usual. Regulars are people who posted in the two months before the campaign; their usual is their mood in the 30 days before it.
• New voices: under the headline, how many new people joined and their share of everyone posting. New voices have no usual, so they aren’t part of the verdict.
• What people talked about: the deal, where new players got stuck, refunds, crashes and performance, game updates, and the game itself, each with its main themes. An update shipped during the campaign is kept apart from talk about the deal.
• Creators: reactions per creator, when you named any.
• Compared with your last campaign of the same type, when you checked one before.
• Mood before, during and after.

Verdicts

Fix check

“Clearly” means the change is bigger than chance could explain. The fewer people complained before the fix, the bigger the drop has to be, so for a small problem a drop to nobody since the fix can still read as Improving rather than Fixed.

Silence isn’t a fix. If the chat got much quieter after the fix, or more than 30% of the people who complained before haven’t posted since, the verdict is Improving at best: they may have stopped playing rather than stopped having the problem.

Balance change

For a nerf or buff, a fix check sorts reactions into four groups: like it or find it fair, miss the old version, say it went too far, and say it didn’t go far enough.

Missing the old version is normal after a nerf and usually fades. From 10 days after the change, when enough people reacted after the first week, the verdict is judged on those later days, and the first week is shown for comparison. If fewer people miss the old version than in the first week, they are left out of the Accepted count, which then only asks whether the people judging the change itself find it fair. So a fading backlash doesn’t count against Accepted, unless it is still 40% or more of the later reactions; then the verdict is Expected backlash.

Release report

Each patch-note item gets its own result once at least 5 people react to it:

Campaign recap

Good to know

• Counts are people, not messages. Someone who posts the same complaint ten times counts once.
• Staff and noisy channels are left out. BetaHub proposes which staff roles to leave out, based on role names such as Developer, Moderator or Community Manager. Check Left out (staff) in the plan and use Change plan if it’s wrong. It also skips channels such as announcements, patch notes, giveaways and bot logs, plus any you name; skipped channels aren’t listed in the plan.
• Steam posts have no author, so each Steam post counts as one person. Steam posts are never counted as regulars, returning players or new voices, and aren’t followed up after a fix. A campaign recap’s verdict therefore rests on Discord only.
• Message text. For privacy, checks read the full text of Discord messages only when full-text processing is turned on for your project. It is off by default, and only BetaHub can turn it on; ask support. Otherwise a check works from the key phrases of each message, quotes show those key phrases, and the report says so under Worth knowing. Steam posts follow the Use post text for Sentiments setting of Steam Discussions.
• The verdict comes from fixed rules on the counts. AI sets up the plan with you, picks the patch-note items to track, reads and labels messages, names themes and writes the summary. It doesn’t decide the verdict, and the summary can only use the measured numbers and real messages.
• A result doesn’t change. It’s saved when the check finishes. To include newer messages, ask again.
• Dates matter. Each platform is measured from its own fix date, and the day the fix shipped isn’t counted. Days 0 and 1 after a release are shown but not scored. Hotfixes count with the release they fix. A fix must be at least 2 days old and a release at least 3 days old; a campaign can be checked while it runs.
• New players are kept apart. When a sale or free weekend brings many new people, a fix check compares only people who were around before, and a release report shows returning players separately.
• Raids are left out of campaign recaps: the same message from 15 or more new accounts within an hour.
• Limits: a project can have up to 5 open checks at once (Setting up, Needs your answer, Ready to run, Queued or Analysing) and start up to 20 checks in 24 hours. Cancelling a check frees an open slot, but it still counts toward the 20.
• Who sees checks: everyone on the project with a role that can view analytics sees all the project’s checks and who asked them, and can answer, run, change or cancel any of them.
• A failed check shows Failed in Your checks. Tell support so we can look at it.

Tips for good questions

• Name the problem the way players do, and the fix: “Are people still complaining about grapple rubber-banding since 1.5?”
• Name both releases when the comparison isn’t the previous one: “How did 1.5 land compared to 1.3?”
• Give the campaign’s name and dates: “How did the Summer Sale (Jun 26 to Jul 10) go?”
• Say when a change is a nerf or buff: “Did players accept the shotgun nerf in 2.3?”

Ask about one problem, release or campaign per check. Community messages show what people said, not sales, conversion or retention.

See Also

• Advanced Features – Sentiment Analysis setup and the sentiment dashboard
• Steam Discussions – Bugs and sentiment from Steam boards
• Developer Dashboard – Your project’s setup guide and, once feedback arrives, an inbox of what needs attention
• Managing Feedback – Releases and their patch notes