Depth of UX
- Concept, design direction, playtesting
- 1 week · August 2026
- Solo, AI-assisted build
Depth of UX is a browser game I built to practise research reasoning. Each fight asks a real research question. You answer in your own words, and the score becomes the damage. Make it through eight fallacies and the HiPPO is waiting at the end.
32
8
0
Argue your way through eight research fallacies, then face the HiPPO.
The fight is the question
Each enemy asks a real research question. The Hearsay Sprite wants to know what is happening when two teammates repeat the same complaint they both heard from the same customer.
A text box and an Attack button. That is the entire interface for combat.
The HiPPO
The final boss cites one customer call that supports their view and ignores your study of fifty users. Two hundred hit points of Highest-Paid Person’s Opinion.
The other enemies are mistakes you might make. The boss is the kind someone else makes at you, and those are harder to answer.
Beat it and the run ends on “The HiPPO Falls: evidence over opinion, at last.”
I was quizzing myself out of my own notes. Then I wondered what would happen if the quiz could hit back.
I keep a vault of UX research notes and had been testing myself the boring way: read a concept, close the file, explain it back. The useful part was producing the reasoning myself. The missing part was wanting to keep going.
“The grading rubric is the combat system. Articulation is the weapon.”
So I made the assessment the mechanic. Every enemy is a research fallacy: hearsay, tiny samples, leading questions, confounds. Every fight asks you to explain what is wrong. Your score becomes damage.
There is no quiz screen anywhere in the game, because the fight is the quiz.
How might a game teach research reasoning by making you explain it back in your own words?
How I built it
I defined the concept, the fallacies, the visual direction, the combat loop, and the playtesting. AI helped me build the TypeScript. I decided how the game should work, caught when the grader was wrong, and chose which failures were worth fixing.
The first grader punished anyone who did not already know the textbook phrase.
The game asks for an answer in your own words. The first grader quietly wanted the rubric’s words instead. A clear plain-language answer could land at 1 damage while the textbook phrase scored well. It was grading vocabulary.
What the first grader did
It compared your words to the rubric’s words
A criterion only counted as hit when at least half its significant words showed up in the answer. So the grader was not measuring whether you understood the fallacy. It was measuring whether you had used its vocabulary.
What that meant in play:
A genuinely well-reasoned answer in plain language scored close to zero
You could know the concept and still lose because the phrasing was different
The failure was silent: the game just said you did 1 damage
Being more generous was not the fix
Version two raised the scores without solving the problem
I added partial credit per criterion, dropped the bar for a hit from 50% word coverage to 34%, curved the low-to-mid range where real answers actually live, and softened the length and hedging penalties. A plain-language answer went from about 1 to 74.
Why that still was not right:
Lenient word-matching still quietly punished anyone who avoided the textbook term
It moved the numbers up without changing what was being measured
Generosity and rigour turned out to need two different mechanisms
Match the idea instead of the vocabulary, and “one person’s story isn’t proof” can score the same as “an anecdote is not data”.
I stopped comparing surface words. The rubric and answer are each reduced to concepts, then compared. The map grew to about 25 groups, including anecdote, evidence, sample, bias, causation, significance, and leading questions.
The synonym map
Two answers, no shared vocabulary, same score
What the map does:
Groups the many ways people actually phrase one idea
Collapses both the rubric and the answer before comparing
Keeps inflected forms in a small recall list instead of stemming every word
The bug that made junk score 66
A stemmer collision I only found by writing a bad answer on purpose
The deliberately weak answer “it is bad, change the words around a bit” scored 66 and falsely matched three rubric criteria. The stemmer was stripping “-ing”; “wording” collapsed into “word” and matched a rubric concept by accident.
The fix:
Strip plurals and “-ed” only, never “-ing”
List inflected forms explicitly in the concept map instead
Found by grading a bad answer deliberately, not by playing well
What it still cannot do
This is lexical concept-matching, not understanding
A genuinely novel metaphor that shares no vocabulary with the concept map can still score too low. I’m naming that limit plainly. The offline grader is the always-available floor, and an optional online grader can handle the harder cases.
How that is handled:
One Grader interface, two implementations behind it
The offline grader is the default and needs no key or network
The model-backed one falls back to it on any error, so the game never stalls
Every number here is measured against the built grader, not estimated.
Once the grader forgave phrasing, it could be beaten by writing no sentences at all.
Making the grader generous opened another hole: a list of keywords could beat a real explanation. I started trying bad answers on purpose before anyone else could.
Two exploits
A keyword dump scored 78. Repeating it forty times scored 92.
Neither answer contained a single piece of reasoning. “Sample size anecdote data evidence severity frequency” beat most genuine attempts, and spamming the same words beat almost everything.
Why it worked:
Concept matching rewards the presence of an idea, not its expression
Nothing stopped the same concept counting again and again
Both were found by adversarial testing, not by normal play
The prose check
Real sentences carry function words. A keyword list does not.
The signal was already sitting in the text. Prose is full of “the”, “is”, “to”, “not”; a keyword dump has almost none. A repetition dump has a very low ratio of unique words. Two ratios, engaging only on answers long enough to judge, were enough to separate reasoning from noise.
What it measures:
Stopword ratio below 0.12 reads as a keyword list and is heavily discounted
Unique-word ratio below 0.30 reads as repetition and is discounted harder
Answers under six words are left alone. They are too short to judge fairly
Where it landed
Forgiving about phrasing. Unforgiving about being right.
Keyword-stuffing dropped from 78 to 39 and repeat-spam from 92 to 20. Genuine answers kept their scores. The ordering matters: terms without reasoning should score below a plain explanation that gets the idea right.
Final calibration, measured:
Empty, off-topic, or junk: 0
Repeat-spam 20 · keyword-stuffing 39
Plain-language real answer 69 · terse but correct 87 · full strong answer 100
200,000 characters graded in 14ms, so no input can stall it
Pick a character, choose a path, and answer in your own words.
Three ways to play
The Analyst reads the data cold and starts every fight already knowing one rubric point. The Field Researcher heals a little at the start of each fight. The Advocate is tougher and shrugs off counterattacks.
The classes work like difficulty settings with personalities. Starting with one rubric point revealed is a genuine hint.
A map you choose through
Fights, elites, campfires and branching text events, generated fresh for every run, with the boss always reachable from the start.
The branching map reshuffles the order, so the same fallacy can arrive in a different context each run.
The most important bug in the project lasted 750 milliseconds.
The rubric breakdown told you which criteria you hit and which you missed. A timer cleared it about 750 milliseconds after it appeared. I found the bug while playing, then changed the state flow so the verdict and rubric chips stay put until the next question.
The verdict and rubric chips persist into the next question · 77 damage, two criteria covered, two missed
The useful bugs showed up when I attacked the game.
I generated 300 to 400 maps and checked them programmatically, fed the grader hostile input, tried to break the JSON, and deployed from a subfolder. That uncovered dead ends, scoring exploits, an XSS risk, and path failures that ordinary play missed.
What I threw at it
Testing the artifact, not the source
Two nasty failures only appeared after deployment: an embedded layout break and a subfolder 404. Both looked fine locally. Testing the shipped artifact turned out to be a separate job from checking the source.
What was tested:
300 to 400 generated maps checked for dead ends and an unreachable boss
Grader fed empty strings, emoji, non-English text and a single 500,000-character token
Nine hostile JSON payloads to the remote grader: all parsed safely or fell back offline
An img onerror payload rendered into the live DOM: HTML escaped, zero tags injected
Network traffic recorded while loading the shipped file: zero requests
What I’d do differently.
- Playtest the balance with other people. A strong answer can one-shot most enemies, and right now the curve is tuned by my own feel.
- Pick a better name before building. “Depth of UX” was a working title that stuck. It does not tell you this is a game or that it covers sampling, confounds, and survey design.
- Take the grader beyond lexical matching. Concept matching handles a lot, but a genuinely new metaphor can still score too low. That is a limit of this approach.
Thank you.