AI did not end the data literacy crisis. It changed it. As machines move from providing information to interpreting it and recommending what to do, the scarce skill is becoming judgment: knowing what to trust, what to question, and what decisions are still ours to make.
Evidence does not make decisions. People interpret it, weigh it, challenge it, and make judgments. A courtroom reveals what leaders can learn about evidence, expertise, confidence, and data-informed decision-making.
Feeling sure doesn't mean you're right. Learn five simple ways to question what feels true, spot what might be influencing you, and make better decisions about what to believe.
The first chart is never the answer. It is the thing that tells you what to ask next. Each ladder below is a common analysis, broken into the questions that climb it.
These are not scripts. They are worked examples of one habit: let each chart provoke the next question, and keep climbing until the answer would change a decision. The rungs name the question, the chart family that answers it, and the thing on that chart that tells you to climb. Your data will skip rungs and add rungs. The habit is the point, not the sequence.
Every ladder below assumes you are holding an analytical question. Most real requests do not arrive that way. They arrive as “how are we doing?” and no chart can answer a question in that form. Two moves get you from there to rung 1.
Move 1 — Sharpen it: from generic to analytical
A question becomes chartable when five things are pinned to it. A measure (what number stands in for the vague word), a population (who or what is included), a comparison (against what: a baseline, a peer, a target, a prior period), a timeframe (over what window), and the stakes (what decision changes depending on the answer). Every ingredient you leave unpinned, the software defaults or the reader's assumptions will pin for you, and usually wrong.
“Are our new hires struggling?” becomes “Is 90-day retention for remotely onboarded hires lower than for in-person cohorts this year, by enough to change the onboarding budget?” Measure, population, comparison, timeframe, stakes. Now a chart can answer it, and you could walk the decision tree with it.
Move 2 — Fan it out: one question contains many
Every sharpened question carries a family of sub-questions, and they fan out along the same few axes every time. Components: what parts make up the number? Segments: for whom, where, which units? Time: since when, how fast, is it accelerating? Reference: compared to what, and is the comparison fair? Cause: what else changed that could explain it? Consequence: what follows if it is true? One question in, six or more out, and each one wants its own chart.
The fan gives you breadth: all the questions living inside your question. The ladders below give you depth: the order to ask them in, so each answer feeds the next. That is the whole relationship between this page and real analysis. Fan for coverage, climb for understanding.
This is the minimum needed to start climbing. Week 5 goes deep on the craft of questioning itself.
General analysis
Performance frameworks
The models below are not new habits. They are the same climb wearing professional uniforms: Kirkpatrick is four rungs, Rummler-Brache is three levels, the Success Case Method is a distribution question. If you can climb the ladders above, you already know how to walk these.
A metric moved, someone is alarmed, and the first chart is pointing at a person or a team. This ladder is how you find out what actually happened before anyone gets blamed.
Running example: a queue metric has climbed from about 30 items a day to over 300 in a quarter. On the trend chart it looks like one person failing. Every rung below is a question that chart cannot answer on its own.
“What exactly moved, and when?”
Before explaining anything, pin down the movement itself. A slow drift, a sharp step, and a one-time spike are three different stories with three different suspect lists. A drift suggests something accumulating. A step suggests something changed on a date. A spike suggests an event that came and went.
Line chart with the change window marked, plotted on real calendar time, not event numbers.
Climb when you can name the shape and the date. The shape narrows the causes, and the date is what you will test everything else against.
“Is the metric moving, or is its denominator?”
Most alarming metrics are ratios or queues, and both have two moving parts. A queue grows when inflow rises or when processing falls, and those are opposite diagnoses. A rate rises when the numerator grows or the denominator shrinks. Plot the parts separately before trusting the whole.
Multi-line inflow and outflow as separate series, or Paired panels of counts and rates together.
Climb when you know which part moved. A tenfold rise in inflow with flat processing is not a performance story, and no amount of staring at the original chart would have told you that.
“Is it everywhere, or somewhere?”
Split the metric by every grouping you have: person, site, product, segment, weekday. Uniform movement points at something systemic, a policy, a season, a measurement change. Concentrated movement points at something local, and names where to look next.
Small multiples one panel per group on identical scales, or a Unit heat map when the grid is big.
Climb when you know the footprint. One unit glowing while the rest sit quiet just converted a company-wide alarm into a specific address.
“What is the mix behind the number?”
Decompose the metric by reason, category, or type. A total is a bundle, and the bundle can hide a single dominant cause. When 96% of a spike carries one reason code, the investigation stops being about the person holding the queue and starts being about whatever generates that code.
Pareto chart when concentration is the question, Stacked bar over time to watch the mix shift.
Climb when one component dominates. The metric was the smoke. The dominant reason is the direction the fire is in.
“What changed in the world when the chart changed?”
Now use the date from rung 1. New system, new staff, new policy, new counting rules, a season, a shortage. Annotate the candidates directly on the timeline and see what lines up. This is also where you leave the data: a conversation or a site visit is sometimes the only way to confirm what the chart can only suggest.
Annotated line chart with candidate events marked as reference lines and shaded ranges.
Climb when one candidate lines up with the date and the footprint. Lining up is not proof, which is exactly why there is one more rung.
“What would prove this explanation wrong?”
Before acting, attack your own answer. If the explanation is an untrained team upstream, then units with trained teams should look normal, and the spike should shrink after training arrives. State the prediction, then build the one chart that could break it. An explanation that survives its own test is worth acting on. One that was never tested is a hunch with a chart attached.
The comparison your explanation predicts whatever chart puts the prediction and the data on one canvas.
Climb when the explanation survives. If it fails, that is not a setback, that is rung 5 handing you a better candidate.
“So what do we do differently?”
The last rung is always a decision, not a chart. Fix the process, retrain the team, correct the metric, or explicitly decide the movement is benign and stop watching it. An analysis that ends without changing anything, a decision, a plan, or at minimum a belief, was decoration. And spare a thought for how it started: the first chart was ready to blame a person. Six questions later, the person turned out to be the one who had been raising the alarm all along.
A result landed against a bar, pass or fail, met or missed, and the room wants a verdict. This ladder is how the verdict becomes a plan.
Running example: a class, team, or process came in at 43% against an 80% bar. The verdict takes one chart. Everything that should happen next takes five more.
“Did we hit it?”
Answer the verdict question first, honestly and fast, because the room will not hear anything else until it is answered. But answer it with the margin attached. Missing by two points and missing by thirty are both a no, and they are not the same no.
Bullet / progress bar actual against the target tick, or a Two-slice donut with a marked majority line when the rule is a majority.
Climb when the verdict is stated with how close. The margin is what decides whether the next rung matters.
“What shape is behind the yes or no?”
The target dichotomized a continuous measure, and the cut discarded the shape. Two groups with identical pass rates can need opposite responses: one has everyone hugging the line, the other has a cluster that never got traction. Dichotomize for the decision, but chart the distribution.
Histogram with a threshold line shares annotated on each side, the two sides colored differently.
Climb when you know where the misses live. Just under the line calls for a nudge. A floor group calls for an intervention.
“Which ones, not how many?”
Aggregates answer allocation questions. Action needs names. One row per unit, colored by where they stand, converts a percentage into a to-do list, and it also exposes whether the misses are the same units every period or a rotating cast.
Unit-level heat map or a Strip plot at small n. Mind privacy: a row is a person.
Climb when the intervention list exists. Counts justify budgets. Names direct effort.
“Are we moving toward the bar, or standing still?”
One period against a target is a snapshot. The trend of the shortfall is the story: a 43% that was 30% last cycle is a different situation from a 43% that was 60%. Put the periods side by side with the target line drawn through all of them.
Diverging stacked by period centered on the threshold, or a Line of the pass rate with the target as a reference line.
Climb when the direction is known. Improving-but-short and declining-but-close get opposite headlines from the same snapshot.
“Is the target right?”
A target is a claim, not physics. Where did the bar come from, whose is it, and when was it set? Compare it against history and against peers. Sometimes the finding is that the group is fine and the number is wrong, and that is a legitimate, defensible outcome of analysis, not an excuse.
Corridor / typical range of historical performance with the target drawn on it.
Climb when the target is either defended or revised out loud. Either way it stops being furniture.
“So what do we do differently?”
Reteach or retrain the floor group, nudge the near-misses, reset an indefensible target, or declare success and stop spending attention. Notice that the rung-1 verdict alone would have chosen the same response for all four of those situations, and it would have been right about once.
Two groups differ on an average, and someone is about to generalize. This ladder is what the gap has to survive before it deserves a conclusion.
Running example: one grade level outscores another on a skills assessment, and the first chart makes the gap look decisive. Five questions decide whether it is.
“How big is the gap, really?”
State the difference in units that mean something to the decision, not just axis distance. A gap that looks dramatic on a zoomed axis can be trivially small in practice, and a modest-looking gap can be enormous. Honest axes first, then judgment about whether the size matters.
Dumbbell per category sorted by gap, or a Grouped bar when there are a few groups.
Climb when the gap has a size in real units and an honest frame. Now it is worth explaining.
“What do the averages hide?”
Two group averages are two numbers standing in for two distributions, and the distributions almost always overlap. Seeing the overlap changes the claim from “group A is higher” to “group A is higher on average, and plenty of individuals run against the pattern,” which is the version that survives contact with reality.
Box plots side by side, Histograms per group, or a Diverging stacked bar for ordinal ratings.
Climb when the overlap is visible. Overlapping distributions forbid sweeping statements about individuals, and that discipline is the point.
“Same units, or different units wearing group labels?”
Before explaining the gap, check what the groups are. This year's cohort versus last year's is different people. Treated versus untreated might differ in ways that predate the treatment. A gap between groups is only a change if the same units moved, and only an effect if the groups were comparable to begin with.
Slope chart when the same units were measured twice, Dumbbell when they were not, and the choice itself is the claim.
Climb when you know whether this is change, difference, or selection. Three different words, three different arguments.
“Does the gap survive a split?”
Cut the comparison by a third variable: site, tenure, level, season. A gap can concentrate in one segment, vanish outside it, or, in the famous case, reverse entirely when the mix differs between groups. A difference that survives its splits is sturdy. One that does not was a mix effect wearing a group label.
Small multiples of the comparison per segment, or a Grouped bar with the split as the grouping.
Climb when the gap holds, moves, or flips. Whichever it does, you now know something the headline number could not tell you.
“Is it stable, or a snapshot?”
One period can be an anomaly: a hard test, a rough quarter, a lucky draw. Track the gap across periods before building anything on it. Persistent gaps deserve structural explanations. Blinking ones deserve patience.
Multi-line one line per group over time, gap annotated.
Climb when the gap has a history. Persistent and stable earns action. Volatile earns another cycle of watching.
“So what do we do differently?”
Intervene on the segment where the gap actually lives, fix the comparison if it was selection, or accept the difference and plan around it. And keep the rung-2 discipline in the write-up: the claim is about distributions, not about every individual in them.
A slice of the work seems to be eating the calendar, and the case has to travel from feeling to finding. This ladder is that trip.
Running example: a portfolio changed shape, the hours no longer fit, and the first chart shows what one demanding unit costs. The argument lives four rungs higher.
“What does one unit cost?”
Establish the per-unit demand with its composition: hours per account, cost per case, minutes per ticket, broken into the tasks that make it up. This is the evidence everything else multiplies from, so get the mix visible now.
Stacked bar per unit type with totals labeled on top.
Climb when the per-unit number and its mix are on the page. Evidence secured. Now build the claim.
“What does the whole portfolio cost?”
Multiply the per-unit cost by the counts and draw the total against capacity. This is the chart that makes the argument, and it is one multiplication away from rung 1, which is exactly why it gets skipped: the analyst does the multiplication in their head and forgets the reader will not. A per-unit chart cannot make a portfolio claim.
Stacked totals old versus new arrangement, with an Available-hours line drawn across.
Climb when the total and the capacity line share a canvas. Above the line is no longer an opinion.
“Is share of effort proportional to share of units?”
The word for this situation is disproportionate, and it is a word people argue with. Replace it with two numbers: this slice is X% of the units and Y% of the effort. Two 100% bars, identical colors, and the mismatch between the colored blocks is the entire argument.
Paired composition bars share of units above, share of effort below.
Climb when the mismatch is quantified. 21% of the accounts taking 58% of the time is a sentence nobody has to interpret.
“Which tasks drive the difference?”
Not everything scales alike. Some tasks cost five times more on the demanding units, some barely double. Ranking the multipliers points the fix at the right activity, meeting cadence rather than paperwork, or the reverse, and turns a workload complaint into a process finding.
Dumbbell / double lollipop per task, sorted by multiplier, the overall ratio drawn as a reference.
Climb when the driver has a name. Reduce the driver, not the workload in general.
“Does the allocation model explain it?”
If assignments are balanced on one measure, revenue, headcount, case count, test whether effort actually tracks that measure. Plot units against the proportional line. Points sitting systematically above it mean the model under-weights their demands, and now the finding is structural: anyone holding these units is overloaded, not just you.
Scatter allocation measure vs actual effort, with the Proportional reference line drawn.
Climb when you know whether the model or the mix is the cause. A structural finding argues for changing the model. A mix finding argues for rebalancing within it.
“So what do we do differently?”
Rebalance the portfolio, add capacity, reduce the driving task, or fix the allocation model itself. The ladder started with someone feeling overloaded and ended with a testable claim about how work is assigned, and that transformation, from complaint to structure, is what makes the case fundable.
Someone is improving and wants to know if it is working, how fast, and what comes next. This ladder answers all three without turning the chart into a leaderboard.
Running example: a novice's times are dropping trial over trial. The first chart celebrates. The next four decide what to do with the celebration.
“Is it moving?”
Establish the trend honestly first: real calendar time so uneven gaps do not distort the rate, the direction of good made unmistakable (invert the axis and label it when lower is better), and units the audience thinks in. A trend chart that needs translation is not yet a trend chart.
Line chart on calendar time, axis direction matched to the audience's sense of good.
Climb when the direction is visible over the noise. If it is not, smooth before concluding, and say what the smoother is.
“How fast, and is the pace changing?”
Improvement is rarely linear. Rebase to percent improvement from the starting point to compare unlike series fairly, and plot the period-to-period change to see acceleration and slowing directly. A positive but shrinking delta means still improving, just slower, and readers will misread that as decline unless you label it.
Indexed line from a common start, and a Rate-of-change line with the zero line drawn strongly.
Climb when the pace has a direction of its own. Speeding up, steady, and slowing are three different coaching conversations.
“Is this trajectory normal?”
Compare the individual to a typical path without building a leaderboard: convert history to percent improvement from each unit's own baseline, take the middle range as a band, and rescale it to this unit's start. The band answers “is my trajectory normal” and is mathematically incapable of answering “am I faster than my peers,” which is the protection.
Banded trajectory (corridor) anchored to the unit's own starting point.
Climb when the trajectory sits inside, above, or below the typical band. Inside is reassurance. Outside, in either direction, is information.
“Where is it improving, and where not?”
One overall trend averages across components that may be moving at very different speeds. Split by component, distance, skill, product line, and gains usually concentrate: quick wins in one place, grinding progress in another. That unevenness is the actionable finding, because effort should go where the curve is still steep.
Small multiples per component on identical scales, or the Indexed lines overlaid.
Climb when the uneven gains are located. Fast components need maintenance. Slow ones need a changed approach, not more of the same.
“What should we expect next?”
Extend the typical band forward across the horizon. Early gains are steep, the bend is coming, and the flattening is physiology or economics, not failure. Showing the plateau before it arrives converts it from a demoralizing surprise into a landmark that was on the map all along. Mark anything projected as projected: dashed beyond today, always.
Long-horizon corridor with the flattening annotated, projections visually separated from measurements.
Climb when expectations are set ahead of the data. The plateau arrives pre-announced, and nobody quits over a chart.
“So what do we do differently?”
Shift effort to the components still climbing, adjust goals to the phase of the curve, and set the next conversation before the bend, not after it. Progress charts are unusual: their job is partly psychological, and a chart that keeps someone going is doing analytical work, provided it never buys motivation with distortion.
Kirkpatrick's four levels are already a ladder, and the field's chronic failure is stopping on rung 1. This ladder climbs all of it, plus the rung Kirkpatrick implies but people skip.
Running example: a program wrapped, the smile sheets are glowing, and someone wants budget for round two. The smile sheets are rung 1 of 5.
“Did they like it?”
Level 1, reactions. Worth collecting, because misery flags delivery problems worth fixing, and worth distrusting, because satisfaction is the weakest predictor of everything above it. Enjoyable programs can transfer nothing, and demanding ones can transfer everything.
Diverging stacked bar of the reaction scale against an acceptability threshold.
Climb when reactions are summarized and treated as a floor, not a verdict. The industry's chronic failure is stopping here, where evaluation is cheapest.
“Did they learn it?”
Level 2. Measure the gain against a pre-measure of the same people, which makes this paired data and licenses a growth claim. And look past the average gain to its distribution, because a class that averaged plus-15 can contain a third who gained nothing.
Slope chart pre to post, same units measured twice, plus a Histogram of gains behind the average.
Climb when the gain is measured against a pre, not assumed from attendance, and the spread of gains is visible.
“Are they doing it?”
Level 3, transfer, where most training evaluation quietly dies. Learning in the room is not behavior on the job, so measure the behavior itself, weeks later, after the glow fades. And when transfer fails, the cause is usually the environment, not the learning, which is the needs-assessment ladder's territory.
Behavior frequency over time observed or system-logged, weeks after the program.
Climb when on-the-job behavior is measured after the honeymoon. If it is not there, everything above this rung is moot.
“Did the business move?”
Level 4. Put the outcome metric on a timeline with the program marked as an intervention line, and set lag expectations out loud, since behavior moves before outcomes do. An unmarked trend invites everyone to credit or blame the wrong thing.
Line chart of the outcome with the Intervention line at the program date.
Climb when the outcome trend and the program date share a canvas, with the lag acknowledged rather than hidden.
“Would it have moved anyway?”
The attribution rung. Extend the pre-program trend forward, or compare against a group that did not get the program. If the metric was already climbing, the program is claiming credit for gravity. Only monetize after this rung: an ROI on unattributed movement is fiction with a dollar sign.
Baseline trend extended or a Comparison group on the same chart.
Climb when a rival explanation has been tested and survived. This is the rung that separates evaluation from advocacy.
“So what do we do differently?”
Keep, fix, or kill the program, and notice which levels the decision actually used. Level 1 data alone answers none of those three, yet it is the level most programs are evaluated at, because it is the level that is easiest to collect. Evaluate where the decision lives, not where the data is cheap.
This ladder runs before any solution exists, and its real job is to stop training from being prescribed for problems training cannot fix.
Running example: results are below expectations and the reflex request has already arrived: build a course. The ladder decides whether a course can even help.
“What is the gap, exactly?”
Actual versus desired, stated in units of performance, not activity. Hours of training delivered is activity. Orders processed correctly per day is performance. If desired has never been defined numerically, that discovery is itself the first finding.
Bullet / progress bar actual against the desired level, per unit or team.
Climb when the gap has a size in performance units. A gap you cannot size, you cannot prioritize.
“Is it worth closing?”
Cost the gap and cost the fix, and compare them honestly. Some gaps are cheaper to live with, and saying so is analysis, not defeatism. This rung exists to kill trivial projects early, before they consume a needs assessment.
Paired bars cost of the gap per period against cost of candidate fixes.
Climb when the stakes are quantified. Trivial gaps exit the ladder here, and should.
“Could they do it if their lives depended on it?”
The Mager and Pipe question, and the fork in the whole model. If yes, it is not a skill gap, and no training on earth will close it. Classify the gap: skill, will, or environment, using observation and interviews, then chart the classification rather than assuming it.
Breakdown by cause category share of the gap attributed to skill, motivation, and environment.
Climb when the cause is classified from evidence. Most performance problems are not knowledge problems, and this is the rung that finds out.
“Which cell of the environment is failing them?”
Gilbert's Behavior Engineering Model: three environment cells (information, resources, incentives) and three individual cells (knowledge, capacity, motives). Work the environment row first, because it usually explains most of the gap and costs least to fix. Binder's Six Boxes walks the same grid.
Pareto chart of barriers coded to the six cells, environment cells flagged.
Climb when the cells are ranked by contribution. Expect the top of the Pareto to be environmental, and be surprised out loud if it is not.
“What mix of fixes matches the causes?”
Map interventions to diagnosed cells: job aids and feedback for the information cell, tools and time for resources, consequences for incentives, and training only where the knowledge cell actually showed up. Every proposed fix should trace to a rung-4 cell, or it is a solution shopping for a problem.
Cross-tab matrix causes down the side, interventions across the top, the mapping visible.
Climb when every fix traces to a diagnosed cause. A training line item with no knowledge-cell evidence behind it gets challenged here.
“So what do we do differently?”
Fix the environment first, train where knowledge is genuinely the gap, and decline to build the course when the diagnosis says the course cannot work. The most common failure in this profession is not bad training. It is good training prescribed for an environment problem, and this ladder is the prescription check.
You changed something on purpose and the metric moved. This ladder decides how much of the movement you are allowed to claim.
Running example: a redesign shipped, the metric is up 12%, and the team wants credit. Some of that 12% was coming anyway. The ladder finds out how much.
“What moved after the change?”
Establish the movement and its timing against the change date. This ladder differs from diagnosing a drop in one important way: there, you are explaining an unexplained change. Here, you made a change on purpose and want credit, which means you are motivated to see success, and the ladder's job is to protect you from yourself.
Line chart with the Intervention line at the change date.
Climb when the movement and its timing are established, and stated neutrally, before any credit is assigned.
“What would have happened anyway?”
Draw the counterfactual: extend the pre-change trend forward, or find a comparison group that did not get the change. The gap between what happened and the anyway-line is the effect. Everything above the anyway-line was coming regardless, and claiming it is the most common sin in change reporting.
Baseline trend extended as a dashed line, or a Comparison group on the same axes.
Climb when the anyway-line is drawn. The honest effect is the distance above it, not the distance above zero.
“For whom did it work?”
Changes rarely work uniformly. Split the effect by segment, site, user type, tenure, and expect concentration. An average lift of 12% that is 30% for new users and zero for veterans is a finding about who the change was really for, and it redirects the next iteration.
Small multiples of the before-after comparison per segment.
Climb when the heterogeneity is mapped. Uniform effects suggest something systemic. Concentrated ones name the real audience.
“Is it lasting, or novelty?”
Re-measure after the honeymoon. Novelty effects fade in weeks, and a launch-window snapshot flatters every change ever made. The decay check is cheap, and skipping it is how organizations accumulate a portfolio of changes that all worked for a month.
Extended timeline through the post-launch window, launch period shaded.
Climb when durability is observed past the honeymoon, not assumed from the launch spike.
“What did it break?”
Check the guardrail metrics: the things the change was not supposed to touch. Speed bought with error, revenue bought with churn, adoption bought with support load. Side effects are real effects, and nobody assigns them a chart unless this rung demands it.
Guardrail panel the two or three at-risk metrics on the same timeline as the win.
Climb when the guardrails are checked and charted, not waved at. A win with an unexamined cost is not yet a win.
“So what do we do differently?”
Scale it, refine it toward the segments where it worked, or retire the claim and admit the trend was coming anyway. Credit is assigned only after the anyway test, and the write-up reports the effect above the counterfactual, which is a smaller and truer number than the one the launch deck wanted.
Survey data is the most common data students bring, and the most commonly over-trusted. This ladder is the distance between the results and the truth.
Running example: the results are in, and leadership wants the number. There is a number. It needs four rungs of context before it means anything.
“Who answered, and who did not?”
Before summarizing the present, name the missing. A 30% response rate means the results describe the kind of person who answers surveys, and nonresponse is rarely random: the furious and the delighted over-respond, the disengaged middle stays silent. The population is defined by who entered it.
Response rate by group and a Response funnel from invited to completed.
Climb when the missing are named before the present are summarized, and the rate rides with every chart downstream.
“What did they say?”
Chart each item at its own measurement level. Ordinal scales go on diverging stacked bars against the threshold that matters. Check-all-that-apply items go on multi-select bars with percent of respondents, never percent of responses. And the overall score, if there is one, ships with its n attached.
Diverging stacked bar for scale items, Multi-select bar for check-alls.
Climb when the results are charted at the right measurement level, with denominators stated. Description done. Now the useful part.
“What drives the overall score?”
Not every low item matters and not every high one is safe. Plot each item's score against its relationship to the overall satisfaction, and the chart sorts items into drivers and passengers. Effort goes to low-scoring drivers. A low-scoring passenger is a distraction wearing an alarm.
Scatter plot item score on one axis, strength of relationship to the overall on the other.
Climb when priorities are ranked by relationship, not by lowest score alone. This is the rung that turns description into a to-do list.
“What do the words explain?”
The numbers say what, the comments say why. Code the open-ended responses and pair them with the items they explain, and check which themes travel together. A falling score with a coherent theme behind it is a finding. One without any words attached is a question for the next survey.
Coded theme counts per item, and a Co-occurrence matrix of themes chosen together.
Climb when every headline number has at least one why attached, or an honest admission that it does not.
“What changed since last time, and for the same people?”
Trend claims need the same-units check: same respondents tracked, or two different samples of whoever answered? The first supports change claims. The second supports comparison only, and drifting response rates can move a score all by themselves with nobody's opinion changing.
Slope chart wave to wave, with the same-sample-or-not stated on the chart.
Climb when change claims are matched to who was actually measured, twice.
“So what do we do differently?”
Act on the low-scoring drivers, take the passenger items off the alarm list, and ship the single number leadership asked for with its base attached: the score, the n, and who is missing. A survey number without its base is a rumor with a decimal point.
Performance lives at three levels, and the cardinal sin is starting at the performer. This ladder enforces the descent order.
Running example: a headline metric slipped and names are already being mentioned in meetings. The ladder makes the names wait until the process has been cleared.
“What does the organization level show?”
Start at the top: the headline metric in its context of goals and environment. Rummler and Brache's three levels, organization, process, performer, are a ladder with a rule attached: you are not allowed to start at the bottom, because the bottom is where the blame is and rarely where the cause is.
Line chart of the headline metric against its goal, annotated with context.
Climb when the organization-level movement is framed. Now descend one level, not two.
“Where in the process is the loss?”
Trace the metric to the process that produces it, and find the step where performance leaks. This is the tree's process branch doing HPI work: a funnel if the population survives stage to stage, a step-rate line if each stage is an independent measure, a Sankey if the paths matter.
Step-rate line, Funnel, or Sankey depending on the process structure.
Climb when the losing step is located. Most performer-level problems are process problems wearing a person's name, and this rung is where the costume comes off.
“Which performers vary, and by how much?”
Only now, inside the losing step, look at individuals. Chart the variation across performers doing the same work, and mind the width: some spread is routine, and a wide spread is a finding about the step, not just its people. Metric alignment matters here too, since a performer measured on speed inside a process rewarded for accuracy is not underperforming, they are obeying.
Strip plot or Unit-level view of performers within the step, spread visible.
Climb when the variation is seen against peers in the same process context, not against a company average that mixes contexts.
“Does context explain the variation?”
Split the performer differences by the conditions they sit in: tools, inputs, sites, shifts, upstream teams. If the low performers share a condition, the condition is the cause and the performers are its evidence. Kaufman's mega-macro-micro asks the same question with a wider top level: sometimes the context that explains everything sits outside the organization entirely.
Small multiples of performer results split by condition.
Climb when person versus context is adjudicated with a chart instead of an opinion. The escalation-queue story from the drop ladder is this rung with a happy ending.
“So what do we do differently?”
Fix the problem at the level that owns it: strategy at the organization level, redesign at the process level, and only after both are cleared, development at the performer level. Changing people to fix a process is the most expensive intervention there is, and the most temporary.
The shortest ladder, and the most protective. Most management reaction is tampering with noise, and this ladder is the tamper-proofing.
Running example: this month's number is down, the meeting is tomorrow, and the room expects a response. The first question is whether anything happened at all.
“What does routine variation look like?”
Every metric wobbles, and the wobble has a normal width. Before judging any single point, draw the range the metric occupies when nothing special is happening, from its own history. Deming's insight is that a system produces variation as a matter of course, and a point inside that range is the system talking, not an event.
Corridor / control band the metric's routine range drawn from history.
Climb when the metric's normal wobble is on the page before this period's number is judged against it.
“Is this point outside the band?”
Compare the alarming number to the band. Inside means the honest conclusion is do nothing, and do nothing is a legitimate analytical result that deserves to be said with a straight face. Reacting to routine variation, praising the up-wobble, coaching the down-wobble, is what Deming called tampering, and it makes systems worse.
The point against the band this period marked on the corridor.
Climb when there is a verdict: common cause, leave it alone, or a candidate signal worth a diagnosis.
“If it is a signal, what is the special cause?”
A true signal earns a real investigation, and the investigation already exists: hand off to the Diagnosing a drop or spike ladder, rung 1, with one advantage in hand. You arrive already knowing the movement is real, which is that ladder's unstated rung zero.
Hand off to the drop ladder, carrying the band as context.
Climb when the diagnosis ladder is engaged for true signals only. Noise never gets that far, which is the entire point of this ladder.
“So what do we do differently?”
React to signals, never to noise. And if the routine band itself sits somewhere unacceptable, the fix is redesigning the system that produces it, not exhorting the people inside it. This is the shortest ladder in the document and the one most likely to save a future manager from themselves.
The Success Case Method is a distribution question wearing an evaluation model's name, and it ladders the way this course keeps teaching: study the shape, then the tails.
Running example: a program's results are middling on average and heading for cancellation. The average is about to make a decision the distribution would veto.
“What is the distribution of outcomes?”
The average says the program half-worked. The distribution says something more useful: usually that some participants transformed, some never started, and the average describes almost nobody. Brinkerhoff's Success Case Method begins where this course keeps ending up, with distrust of the mean.
Histogram of participant outcomes, with the average marked and mistrusted.
Climb when the shape is visible. Two humps mean two stories, and the rest of the ladder studies the humps, not the middle.
“Who are the extremes?”
Identify the highest and lowest success cases by name, deliberately and in small numbers. This is the method's signature move: instead of surveying everyone shallowly, study the tails deeply. The extremes hold the information, because they are where the program met the most and least favorable conditions.
Unit-level view sorted by outcome, top and bottom cases flagged.
Climb when the cases are named for study, not just counted. Privacy discipline applies: a row is a person.
“What is different about the successes?”
Compare the conditions around the winners and the non-starters: manager support, opportunity to apply, tools, workload, timing. This is Gilbert's environment cells applied to the tails, and the finding is almost always that success had conditions, not just participants. Verify with interviews, because the chart can locate the difference but not explain it.
Condition comparison success factors present in top cases versus bottom, side by side.
Climb when the success conditions are identified, as distinct from the success people. Conditions can be spread. People cannot.
“What is the unrealized value?”
The method's closing argument: if the median participant performed like the top quartile, what would that be worth? That gap is the program's unrealized value, and it is chartable as a simple, confronting comparison. It reframes the middling average from a verdict on the program into a measure of how much the conditions are leaving on the table.
Paired bars current overall result against the result if typical matched the top quartile.
Climb when the gap between typical and demonstrated-possible is quantified. That number is the budget argument.
“So what do we do differently?”
Spread the conditions the successes had, remove the barriers the non-starters hit, and stop rerunning the training unchanged, because the training already worked, for the people whose environment let it. The average told you the program half-worked. The extremes told you why, and which half to fix.
Descriptive, then diagnostic, then structural, then a decision. Rung 1 establishes what happened. Rungs 2 through 4 locate it: which component, which unit, which mix. Rungs 5 and 6 explain it and test the explanation. The last rung changes something. Different analyses climb different ladders, but they all climb in that order, and the most common failure in real organizations is not a wrong chart. It is stopping on rung 1.
For the HPI-minded: Rothwell's model is the master ladder. Performance analysis, cause analysis, intervention selection, implementation, evaluation. The needs-assessment ladder is its opening rungs, Kirkpatrick and attribution are its closing ones, and everything in between is cause analysis done with charts. The frameworks were never separate from the habit. They are the habit, written down by people who climbed it for a living.
