How to evaluate an AI gameplay commentator: relevance, timing, repetition and silence.

The method in one paragraph

Pick one recorded session in a game you know. Watch ten minutes without skipping. Each time the Broker speaks, note what was on screen and judge four things: was the line relevant, was it on time, was it a repeat, and when he said nothing, was the silence right? Then run the same ten minutes on your own game with the free trial. Four columns, one sheet, your own verdict. This page gives you no scores, because nobody here has scored the recordings for you, and a number from the maker would not be worth much anyway.

1. Relevance: does the line match the screen?

Pause the video the moment a line ends and look at the frame from a few seconds earlier. Does the remark refer to something that was actually there: the object, the character, the outcome? A relevant line can still be wrong about why it happened; what you are checking first is whether he was looking at the same thing you were.

• Counts as a hit: the line names or reacts to something visible or just finished.
• Counts as a miss: the wrong object, the wrong outcome (a win called a loss), or generic filler that would fit any game.
• Mark separately: lines that are relevant but trivial. They are not misses; they tell you about his taste, which comes under “the character”.

2. Timing: does the comment still apply when it lands?

Every line takes time: he looks, the service thinks, the voice is produced, the audio plays. The question is not whether there is a delay, but whether the moment is still alive when the line arrives. A reaction to a crash two seconds later is fine; a reaction to a crash after you have already respawned and driven off is stale.

• Note the gap between the event and the first word. You do not need a stopwatch; “immediate”, “a beat late”, “stale” is enough.
• Watch for lines that interrupt a new moment while describing an old one. Those cost more than silence.
• Fast games (a roguelike, a shooter) expose timing hardest; slow games (strategy, management) hide it. Judge the game you actually play.

3. Repetition: the same observation twice

Count repeats per ten minutes: the same point made again, the same phrase reused, the same joke. One repeat in ten minutes is human; a pattern is not. Also note verbal tics that recur without carrying information.

Distinguish repetition from continuity. Referring back to something that happened earlier in the session (“that is the second time that door got you”) is memory, and it is one of the things a cohost is for. Saying the same thing about the same door with no awareness that he already said it is repetition.

4. Silence: was the quiet right?

Silence is a decision he makes: he holds during cutscenes, dialogue and menus, and when nothing is worth a line. On the Broker Bar that state reads LISTENING. So a quiet stretch is not automatically a failure, and you should judge it against what was happening.

• Right silence: a menu, a loading screen, a cutscene, a long stretch of nothing in particular.
• Wrong silence: a clear moment passed unremarked while he was LISTENING, not paused.
• Not silence at all: VOICE RECOVERING on the Bar means his voice is reconnecting while he keeps watching. That is a service state, not a judgement, and it should be noted as such.

In a recording you cannot see the Bar, so judge silence by what was on screen. In your own session you can.

Three worked examples, from the recordings

These rows show how the sheet is filled in. Each line was read from the Broker’s speech bubble in the overlay of a public recording and timed on the audio-derived captions; what was on screen was read from the same frames. They are three moments chosen to show one hit, one miss and one stale line. They are not a score for any recording, build or game, and they say nothing about how often each happens.

So Leeds nudges ahead just before the half … right now it’s Leeds’ party

that Ruby Key’s a freaking ten grand ticket … Three grand spare, though

this big guy looks like trouble … shiny blue armor

What the rows teach. Relevance and timing are separate columns: the armour line is right about the enemy and wrong about the moment, so it gets a hit in one column and “stale” in the other. A miss is recorded against what the screen showed, not against how the match felt. And the same kind of reading, a scoreboard, lands as a hit at 6:00 and a miss at 8:28, which is why the method asks for ten minutes and not one line. For more on why a screen can be read wrong, see what screen-based commentary can misunderstand.

And the character

After the four columns, there is the one that cannot be tabulated: do you like having him there? Dry observations, an unexpected line, room for silence are the intent. The SPECTATOR role comments from outside; the PLAYER role speaks as if sitting beside you, which changes the tone of the same observation. Talkativeness changes how much of all this you get. None of that is measurable, and it decides everything.

Run it on your own game

1. Claim the trial. 30 Signal Minutes (exactly 1,800 Signal Seconds) through your eligible account; no card, no automatic subscription. Signal Time measures his work, not the clock, so a ten-minute test does not spend ten minutes.
2. Use the game you know best. You can only judge relevance where you know what is relevant.
3. Pick one role and one Talkativeness level and keep them for the whole ten minutes; change one thing at a time between runs.
4. Record it (OBS, with Desktop Audio, as in the OBS guide) so you can re-watch the moments instead of judging from memory.
5. Fill in the sheet while watching the recording, not while playing.

Founder Alpha is early-access software; the commentary can be wrong, repeated or badly timed, some anti-cheat protected games refuse capture, and Founder Alpha is sold to customers in the United States only.

The sheet, printable

Download the scorecard as plain text (twelve rows, the silences box, totals and the vocabulary above; it prints on one page at a normal font size) or copy the table below into any notes app.

Header

Date, game, build (the app shows it under Help/About), role (SPECTATOR or PLAYER), Talkativeness, and whether the source was a recording or your own session.

One row per line

Time, what was on screen, what he said in a few words, then the three judgements and whether you would keep the line.

Silences

Only stretches longer than a minute; mark VOICE RECOVERING separately, because it is a service state, not a decision.

If you publish your sheet, say which build you used, and which role and Talkativeness level, so a reader can compare like with like.

Limits

• This article gives a method and a sheet; it reports no results, ratings or benchmark numbers for any build or game.
• The worked examples are single moments chosen to illustrate the columns; they are not a score for any recording, build or game, and no accuracy rate is claimed.
• The recordings were made on earlier Founder Alpha builds; behaviour changes between builds.
• No guaranteed cadence: how often he speaks depends on the game, the moment and the service.