What this reference is for
Most students arrive at a dataset with one question already in mind. They run that question, get a number back, and stop. The question seemed like the right one to ask, so they trusted it. They didn't generate alternatives. They didn't expand. They didn't ask the questions that would have changed the answer.
This is not a thinking failure. It is a habit gap. Generating multiple questions before settling on one is a learnable skill, and most people are not taught it.This resources teaches you a ladder with twelve question categories to let you check whether you have asked enough kinds of questions. Eight generative moves let you take any starting question and turn it into a family of related questions worth asking.
Use the Overview to see the case for question expansion. Use the Categories tab to make sure you have not missed an entire kind of question. Use the Moves tab to take one question and generate variants. Use the Generator tab as a quick lookup when you are stuck.
Why most analyses stop too early
Watch any group of people working on a dataset and you will see the same pattern. Someone asks a starting question. The group runs it. A number comes back. The group writes the number down and moves on.
The starting question almost always sits at one level, looks at one comparison, covers one time window, includes everyone, and stops there. The question is not wrong. It is just one of many questions the data could answer, and the group never generated the others. The conclusions they walk away with rest on a question they did not test against alternatives.
A common pattern
A team is asked to evaluate whether a new training program is working. Their starting question: "Did completion rates go up after we launched the program?"
They run the number. Completion rates went up by 6%. The team reports back: yes, it is working.
What they did not ask: Did it go up for everyone, or only some learners? Did it go up at the expense of something else? Was 6% a real change or a normal fluctuation? Compared to what historical pattern? What would we have expected without the program? Who is excluded from the data? At what point would we have called it a failure?
Each of those questions could have changed the recommendation. The team did not skip them on purpose. They just did not generate them.
What this companion is and is not
This is not a list of questions to copy. The questions you ask have to be specific to your data, your context, and your purpose. A pre-written list would let you skip the thinking, which defeats the point.
This is a generator. The categories give you a checklist of question kinds so you can spot what you have not yet asked. The moves give you mechanics for turning one question into many. Together they help you build the habit of asking more questions before you settle on one.
The two halves of the companion
Question Categories: a checklist for breadth
Twelve categories of question that tend to come up in data work. Different kinds of question reveal different kinds of insight. If you have only asked descriptive questions, you have only seen one face of the data. The categories help you check whether you have covered enough different angles.
01
Descriptive
What is the case right now?
02
Comparative
How does X compare to Y?
03
Trend & change
How is this moving over time?
04
Distribution
How is this spread? Where are the extremes?
05
Relational
What moves with what?
06
Composition
What makes up the whole?
07
Subgroup
Does this hold for everyone?
08
Threshold & tipping
At what point does this change?
09
Counterfactual
What would have happened if?
10
Causal & mechanistic
Why is this the case?
11
Data quality & provenance
Can the data answer this at all?
12
What's missing
Who or what is excluded?
Generative Moves: mechanics for depth
Eight moves that take any starting question and produce a variant worth asking. Use them when you have a question already and want to find the questions next to it.
Move 01
Shift the level
Move up or down the ladder.
Move 02
Shift the time window
What if you used a different period?
Move 03
Swap the comparison
Different baseline, different answer.
Move 04
Ask the inverse
Flip the question on its head.
Move 05
Ask the threshold version
At what point would this matter?
Move 06
Ask about subgroups
For whom does this hold?
Move 07
Ask the data quality version
Can the data even answer this?
Move 08
Ask what would change your mind
What evidence would flip your view?
How to use this in practice
You don't have to ask every question in every category every time. The point is to be deliberate about which questions you ask and which you leave out, rather than defaulting to the one that came to mind first.
For a quick scan (under five minutes), pick three categories you usually skip and ask one question from each. Even three new angles will shift the analysis.
For a full pass (twenty to thirty minutes), walk through all twelve categories before you start running numbers. Write down the questions you generate. Pick the four or five worth pursuing.
When you are stuck, take your current question and apply the eight moves in order. At least three of them will produce a question you should be asking.
The skill this builds: not asking better questions in the moment, but building the habit of generating more questions before settling. Over time you will internalize the categories and moves and the generation will happen automatically.
Twelve question categories
Each category covers a different kind of question worth asking about data. The categories are not mutually exclusive. Most real analyses pull from several. The goal is not to ask one question from each, but to scan the list and notice which categories you have not engaged with.
Each card has the category's purpose, when to use it, a set of question stems you can adapt, and a worked example using the same recurring scenario: a fictional company evaluating whether a new training program is improving employee performance. Watching one scenario through twelve question lenses shows how dramatically the answer can shift depending on which question you ask.
Purpose: establish what is. Get the basic facts of the data on the table before doing anything else with them.
When to use: at the start of every analysis. You cannot ask any other category of question well until you have a clear descriptive baseline.
Question stems
- What is the [metric]? The simplest descriptive starting point.
- How many [units] are there? Counts and totals.
- What does the typical [unit] look like? Central tendency.
- What is the range of [metric]? Minimum to maximum.
- When does [event] usually happen? Timing and frequency.
In our scenario
"What is the current average performance score for employees who completed the training, and how does that compare to those who did not?"
Purpose: put one number next to another so the first one becomes meaningful. A number alone tells you nothing about whether it is good, bad, normal, or surprising.
When to use: any time you have a single number and need to know whether it matters.
Question stems
- How does [group A] compare to [group B]? Side-by-side comparison.
- How does this compare to [historical baseline]? Compared to past performance.
- How does this compare to [benchmark]? Compared to an external reference.
- How does this compare to what we expected? Compared to a target or forecast.
- How does this compare to [similar context]? Compared to a peer or analog.
In our scenario
"How does the post-training performance score compare to performance scores from this same time of year in previous years, before the program existed?"
Purpose: see how the metric is moving over time, which is different from seeing where it is right now. A high number that is dropping fast is a different story from a low number that is climbing.
When to use: whenever the data has a time dimension, which is most of the time.
Question stems
- How has [metric] changed over [time period]? Direction and magnitude.
- Is the trend accelerating, decelerating, or steady? Second-order change.
- When did the change start? Identifying inflection points.
- Is this a one-time shift or an ongoing pattern? Step change vs. gradual trend.
- What was the trajectory before [event]? Establishing the counterfactual baseline.
In our scenario
"How has performance score changed over the twelve months before and the twelve months after the training program launched, and is the trend continuing or flattening?"
Purpose: see how values are spread, not just where the middle is. The average can stay the same while the spread changes dramatically. Outliers and skew often matter more than the central tendency.
When to use: whenever you have used an average or total, ask the distribution question to check whether that single number is hiding important variation.
Question stems
- What is the spread of [metric] across [units]? Range and variability.
- How concentrated or dispersed is [metric]? Tight cluster vs. wide spread.
- Where are the outliers? Extreme values worth investigating.
- Is the distribution skewed? Long tail or symmetric.
- What percent of [units] fall above/below [threshold]? Cumulative breakdown.
In our scenario
"What does the full distribution of post-training performance scores look like? Is the average being driven up by a small number of high performers, or is the whole distribution shifting?"
Purpose: see what moves with what. Relationships between variables let you understand the system rather than just observe one piece of it.
When to use: when you have multiple variables and want to know whether they connect. Note that relational does not mean causal. Two things can move together without one causing the other.
Question stems
- Does [A] correlate with [B]? Basic association.
- When [A] changes, what happens to [B]? Directional movement.
- Which variables move together, and which move opposite? Mapping the variable system.
- Is the relationship linear, nonlinear, or step-shaped? Functional form.
- Does the relationship hold across all groups, or only some? Conditional relationships.
In our scenario
"Does the size of the performance change correlate with how much of the training each employee completed? Or does completing 30% give the same gain as completing 100%?"
Purpose: see what makes up the whole. A total tells you the volume; the composition tells you what kinds of things are in it.
When to use: when you have an aggregate (total, sum, average) and want to know what is driving it. Composition questions often reveal that "the number" is mostly one thing, not many things.
Question stems
- What does [aggregate] consist of? Breakdown into parts.
- Which [categories] make up the largest share? Concentration.
- Has the composition changed over time? Compositional shift.
- If we removed [category X], what would the answer be? Sensitivity to one component.
- What share comes from the top [N] [categories]? Pareto-style concentration.
In our scenario
"Of the total performance gain attributed to the program, what share comes from each department? Is the result driven by one team's results, or distributed?"
Purpose: check whether the overall pattern holds for everyone, or whether some groups are dragging the average around. The aggregate often hides important differences.
When to use: any time you have an average or total computed across multiple groups. Subgroup questions are how you find Simpson's Paradox before it embarrasses you.
Question stems
- Does the pattern hold for [group A] as well as [group B]? Universality check.
- Which subgroups show the strongest [effect]? Identifying who is moving.
- Which subgroups show the opposite [effect]? Finding the contradictions.
- Are some subgroups missing from the data entirely? Coverage check.
- If we look only at [group X], does the conclusion change? Subset sensitivity.
In our scenario
"Does the performance gain hold for new hires as well as experienced employees? For all departments, or only some? For employees who chose the training, or also for those who were assigned to it?"
Purpose: identify the points at which behavior changes. A small change near a threshold can have outsized consequences. A large change far from one may have none.
When to use: when the metric you care about has known break points (a passing grade, a profit margin, a service level). Also when you suspect the relationship between variables is not smooth.
Question stems
- At what value does [metric] cross the line that matters? Identifying the threshold.
- How many [units] are within [X] of the threshold? Risk of crossing.
- What size change would move us across the threshold? Sensitivity to small movements.
- Does behavior change at any specific value? Detecting break points in the data.
- What is the minimum effect size that would matter? Practical significance.
In our scenario
"At what completion rate does the training start producing meaningful performance gains? Does someone who completes 50% see roughly half the benefit, or none until they finish?"
Purpose: compare what happened to what would have happened otherwise. The question "did things improve?" is often less useful than "did things improve more than they would have without our intervention?"
When to use: when evaluating an intervention, decision, or change. Without a counterfactual, you cannot tell whether the thing you did is the reason for the outcome.
Question stems
- What would have happened without [intervention]? The basic counterfactual.
- Compared to a comparable group that did not [receive intervention], what is the difference? Control comparison.
- Was the trend already moving this direction before [event]? Pre-existing trajectory.
- What other things changed at the same time? Confound check.
- If we had done [alternative], what might have happened? Comparing decision options.
In our scenario
"Was performance already trending up before the training program started? What would performance look like today if the program had never existed and the prior trajectory had continued?"
Purpose: understand why the pattern exists, not just that it exists. Description tells you what is. Causation tries to tell you why, which is harder and almost always more useful.
When to use: when you need to act on the finding. You can describe a pattern without knowing its cause, but you cannot reliably change a pattern you do not understand.
Question stems
- What plausible mechanism could produce this pattern? Generating hypotheses.
- What other explanations are consistent with this data? Alternative explanations.
- What would we expect to see if [mechanism A] were true? Testing implications.
- What evidence would distinguish between [mechanism A] and [mechanism B]? Falsification.
- Could the relationship run the other direction? Reverse causation.
In our scenario
"If the training is causing performance gains, what specifically is the mechanism? Better skills? Better confidence? Better visibility to managers? Each mechanism implies a different next intervention."
Purpose: ask whether the data can answer the question at all. The right question on the wrong data produces a confident-sounding wrong answer, which is worse than no answer.
When to use: at the start of any analysis, and again whenever you find a result that surprises you. Surprising results are often data problems before they are real findings.
Question stems
- Where did this data come from? Provenance.
- How was it measured or recorded? Measurement procedure.
- What definition of [variable] is being used here? Operational definitions.
- How current is this data, and does the lag matter? Recency.
- What known errors, gaps, or inconsistencies exist? Documented limitations.
In our scenario
"How is performance score actually measured? Self-rated? Manager-rated? Outcome-based? Did the measurement method change at the same time as the training program launched?"
Purpose: ask what is excluded from the dataset. Every dataset has a coverage boundary. Pretending the data covers everyone in scope leads to confident conclusions about people the data has never seen.
When to use: before reporting any conclusion that will be applied to real people or real systems. Especially important when stakes are high or affected groups are unevenly represented.
Question stems
- Who or what is excluded from this dataset? Coverage gaps.
- Whose [behavior, outcome, voice] is not captured? Whose perspective is missing.
- What events are not recorded because they were not measured? Unobserved phenomena.
- What time periods are not covered? Temporal gaps.
- Who chose to participate, and who did not? Self-selection.
In our scenario
"Whose performance is missing from the dataset? Employees who left mid-program? Employees in roles where the metric is not tracked? Contractors? People who declined to participate? The training might look effective only because the unsuccessful participants disappeared from the analysis."
Eight generative moves
Each move takes one starting question and produces variants. The categories tell you whether you have asked enough kinds of questions. The moves give you mechanics for taking any single question and finding the related questions next to it.
Use these when you have a question already and want to expand it. Apply each move and at least three will produce a variant worth considering.
The mechanic: if your starting question is at one level, ask the same question one level higher (more abstract, more aggregated) and one level lower (more specific, more granular). The high-level version often connects to strategy. The low-level version often connects to mechanism.
Worked example
"Did completion rates go up after we launched the training program?"
- Higher level: "Is our overall capability development strategy working as intended?"
- One level up: "Has employee engagement with all our learning programs changed?"
- One level down: "Did completion rates go up for the first three modules of the program specifically?"
- Lowest level: "Which individual employees are completing module 3 within seven days, and which are stalling?"
The mechanic: ask the same question for a longer window, a shorter window, a different window, or the window before the one you are looking at. A six-month gain might be a six-week reversal. A quarterly improvement might disappear at the annual level.
Worked example
"Did completion rates go up in the first quarter after launch?"
- Longer window: "Did completion rates go up over the first full year?"
- Shorter window: "Did completion rates go up in the first month, the second month, the third month? Is the gain accelerating or decelerating?"
- Different window: "What did completion rates look like in the same quarter last year, before the program?"
- Pre-launch window: "What was the trajectory of completion rates in the year before launch? Was it already trending up?"
The mechanic: any comparative claim depends on what you are comparing against. Try at least three different baselines. The result that is impressive against one baseline is often unimpressive against another, and the choice of baseline is often what determines whether you see a story or not.
Worked example
"Completion rates went up by 12% after launch. Is that good?"
- Compared to historical baseline: "How does 12% compare to year-over-year changes in completion rates from prior years, before the program?"
- Compared to similar groups: "How does our completion rate change compare to companies of similar size and industry?"
- Compared to expectations: "What did we forecast the gain would be? Is 12% above or below what we expected?"
- Compared to alternatives: "What completion gains did we see when we tried other interventions in the past?"
- Compared to no change: "What random fluctuation in completion rates is normal quarter to quarter? Is 12% outside that range?"
The mechanic: the original question often hides assumptions about the right direction to look. The inverse forces those assumptions out. If you asked "who completed?" also ask "who didn't, and why?" If you asked "what worked?" also ask "what didn't, and what does that tell us?"
Worked example
"Who completed the training program, and how did they perform afterward?"
- Inverse 1: "Who did not complete the program, and why? What is happening in their work that prevented completion?"
- Inverse 2: "Who did the program reach but fail to engage? Who started and dropped out?"
- Inverse 3: "Who improved in performance even without completing the program? What can we learn from them?"
- Inverse 4: "Who completed the program but did not improve? Why didn't the training transfer for them?"
The mechanic: turn vague evaluation into numerical commitment. Instead of "did it work?" ask "at what gain would we call it a success, and at what gain would we call it a failure?" The thresholds force you to commit to standards before you see the result.
Worked example
"Was the training program successful?"
- Success threshold: "What size performance gain would we have needed to see to call this program a clear success?"
- Failure threshold: "What result would have caused us to shut the program down or redesign it?"
- Practical significance: "What is the minimum effect size that would actually change a manager's decisions?"
- Cost-justification threshold: "How large does the gain need to be to justify the cost of running the program?"
- Sustained-result threshold: "How long does the gain need to persist for it to count as a real effect rather than a novelty bump?"
The mechanic: identify the subgroups that matter for your analysis (department, tenure, role, region, demographic) and ask the original question for each. The pattern that holds in the aggregate may reverse for some subgroups, hold strongly for others, and not appear at all in a third.
Caution: resist the temptation to slice into so many subgroups that you find a positive result by chance. Pick subgroups that have substantive reasons to differ before you start cutting the data.
Worked example
"Did training completion improve performance overall?"
- By tenure: "Did it improve performance for new hires? For experienced employees? For long-tenured employees?"
- By department: "Did the gain hold across all departments, or only some?"
- By role: "Did individual contributors gain as much as managers? Were the gains different for technical vs. customer-facing roles?"
- By baseline performance: "Did the program help underperformers more than top performers? Or the reverse?"
- By participation type: "Did self-selected participants gain more than assigned participants? What does that tell us about motivation effects?"
The mechanic: for any question you plan to answer, ask three companion questions. What is the source of this data? How is the metric measured? What is excluded? Each one can flip the conclusion if the answer reveals a problem.
Worked example
"Did training completion improve performance scores?"
- Source check: "Where do these performance scores come from? Manager assessments? Self-report? Outcome data? Each source has different reliability."
- Measurement consistency: "Did the way performance was measured change at the same time the program launched?"
- Definition stability: "Is 'completion' defined the same way it was a year ago, or did the definition shift?"
- Coverage: "Are all employees included in the performance dataset, or only those who got assessed? Who got skipped?"
- Recency: "How current is the performance data? Are we measuring lag effects or current state?"
The mechanic: for any conclusion you are forming, ask what observation would force you to revise it. Then ask whether your data has the power to detect that observation. This move is the strongest defense against motivated reasoning, because it forces you to commit to falsifiable standards before you finish analyzing.
Worked example
"I think the training program is working."
- What would change my mind on direction: "What pattern in the data would convince me the program is not actually causing the gains I am seeing?"
- What would change my mind on size: "What evidence would suggest the gains are smaller than they appear, or are concentrated in a way that makes them less meaningful?"
- What would change my mind on durability: "What signal would tell me the gains are temporary rather than lasting?"
- What would change my mind on cause: "What other explanation, if true, would account for the same data without the program being the driver?"
- What I have not yet checked: "Of the things that would change my mind, which have I tested? Which am I avoiding?"
Building the habit: Apply all eight moves to your starting question, in order, the first few times you use this companion. After a while, the moves become automatic and you will generate variants without consulting the list. The point of the companion is not to be a permanent reference. It is to teach a habit that eventually runs on its own.
Quick Reference
The twelve categories with one stem each, condensed for fast lookup. Use this to spot what you have not asked.
| # |
Category |
Sample stem |
When to use |
| 1 |
Descriptive |
What is the [metric]? |
At the start of every analysis. |
| 2 |
Comparative |
How does this compare to [baseline]? |
Whenever a single number needs context. |
| 3 |
Trend & change |
How has [metric] changed over [period]? |
Whenever the data has a time dimension. |
| 4 |
Distribution |
What is the spread of [metric]? |
Whenever you have used an average or total. |
| 5 |
Relational |
Does [A] correlate with [B]? |
When you have multiple variables. |
| 6 |
Composition |
What does [aggregate] consist of? |
When you have a total or sum to break down. |
| 7 |
Subgroup |
Does the pattern hold for all [groups]? |
Whenever you have an aggregate across groups. |
| 8 |
Threshold & tipping |
At what value does behavior change? |
When break points or minimums matter. |
| 9 |
Counterfactual |
What would have happened without [intervention]? |
When evaluating decisions or interventions. |
| 10 |
Causal & mechanistic |
What mechanism could produce this pattern? |
When you need to act on the finding. |
| 11 |
Data quality & provenance |
How was this measured? |
At the start, and after surprising results. |
| 12 |
What's missing |
Who or what is excluded from this dataset? |
Before reporting any conclusion. |
The eight generative moves at a glance
The mechanics for taking one starting question and turning it into a family of related questions. Apply each move to your starting question; at least three will produce a variant worth asking.
| # |
Move |
The mechanic |
| 1 |
Shift the level |
Ask the same question one level higher and one level lower on the ladder of abstraction. |
| 2 |
Shift the time window |
Ask for a longer window, a shorter window, a different window, or the window before this one. |
| 3 |
Swap the comparison |
Try at least three different baselines: historical, peer, expectation, alternative. |
| 4 |
Ask the inverse |
Where the original asks about success, ask about failure. Where it asks about presence, ask about absence. |
| 5 |
Ask the threshold version |
What value counts as success? What value counts as failure? Commit to thresholds before seeing the result. |
| 6 |
Ask about subgroups |
Take the aggregate question and ask whether it holds for each meaningful subgroup. |
| 7 |
Ask the data quality version |
Source, measurement, definition, coverage. Can the data support the question? |
| 8 |
Ask what would change your mind |
What evidence would force you to revise your conclusion? Have you checked for it? |
Worked example: one question becomes twelve
What does it look like when you actually apply the categories and moves to a single starting question? Below is one example, walked through.
The setup
A team is asked to evaluate a new sales coaching program
The team has six months of data since the program launched. Their starting question is the obvious one. Watch what happens when they apply the categories and moves.
"Did the sales coaching program improve performance?"
Twelve variants generated by walking the categories
Descriptive
"What is the average sales performance for participants vs. non-participants right now?"
Comparative
"How does the post-program performance compare to performance during the same six-month window in previous years?"
Trend
"Were sales already trending upward in the months before the program launched?"
Distribution
"Is the average lifted by a few star performers, or did the entire distribution shift?"
Relational
"Does the size of the performance gain correlate with how many coaching sessions someone attended?"
Composition
"Of the total revenue gain, what share comes from new accounts vs. existing accounts vs. expansion?"
Subgroup
"Did the gain hold for new sellers and tenured sellers? For different territories? For inbound vs. outbound roles?"
Threshold
"At how many coaching sessions do we start seeing meaningful performance gains? Is there a minimum dose?"
Counterfactual
"What would performance look like today if the program had never launched and the prior trajectory had continued?"
Causal
"If the program is causing the gain, what's the mechanism? Better skills? Better confidence? Better visibility to managers? Each implies different next steps."
Data quality
"How is sales performance measured here? Did the measurement method or the territories or the quotas change at the same time the program started?"
What's missing
"Whose performance is missing from the data? Sellers who left? Sellers who declined to enroll? Sellers in roles where revenue isn't tracked the same way?"
Five additional variants from applying the moves
Move: Inverse
"Who participated in the program but did not improve, and what does that tell us about who the program is for?"
Move: Time window
"What did the gain look like in month 1, month 3, month 6? Is the effect growing, flat, or fading?"
Move: Threshold
"What size effect would have caused us to discontinue the program? Have we hit it? Are we close to it?"
Move: Swap comparison
"How does our coaching program's effect compare to industry benchmarks for sales training programs?"
Move: Change my mind
"What pattern in this data, if I saw it, would convince me the program is not actually working? Have I looked for it?"
From one starting question, this team now has seventeen questions worth considering. They will not run all seventeen. But picking the four or five most relevant from a set of seventeen produces a much better analysis than running the first one that came to mind.
The expansion process
A repeatable process for taking any starting question through expansion. Use this until the moves become automatic.
Steps to expand a question
- Write your starting question down, exactly as you would have asked it before reading this. This is your default question.
- Walk the twelve categories. For each, ask whether your starting question covers it. If not, generate one variant in that category.
- Apply the eight moves. Take your original question and apply each move in order. Some moves will not apply; that's fine. Skip them and move to the next.
- Review what you generated. You should now have somewhere between ten and twenty questions related to your starting one. Read them all.
- Identify the questions you cannot ignore. Mark the ones that, if the answer turns out a certain way, would change your conclusion. These are the must-ask questions, regardless of how interesting they sound.
- Pick the four or five worth pursuing. Combine the must-ask questions with two or three others that round out the picture. Acknowledge which questions you are deliberately not pursuing, and why.
- Run the analysis. Now you can.
The point of the process: not to add work, but to redirect work. The same hours you would have spent running one analysis go further when those hours are spread across questions that test each other. Expansion takes a few minutes up front and saves hours of revisiting later.
