DAP Scenario Practice — Measurement Foundations

Their Goal

Improve student resilience and psychological wellbeing so that students recover more effectively from academic and personal setbacks.

Their Theory of Change

Students lack coping skills and awareness of available support. If we expose more students to wellness programming, resilience will increase.

Their Plan

Launch a six-week "Resilience Skills" workshop series, run three times per semester. Partner with Residence Life for promotion. Offer a digital resource library. Provide drop-in counseling hours twice weekly.

Their Evidence Plan

Track total workshop attendance. Track number of unique students served. Track digital resource library page views. Report these numbers at end-of-semester review.

End-of-Semester Report — What They Said

"The Resilience Initiative reached 312 students across 18 workshop sessions, with a 94% satisfaction rate and 4,200 digital resource page views — exceeding our targets in every category." The director presented this to the Dean as evidence the initiative was working. The Dean approved budget renewal.

Group Diagnosis — Round 1

Discussion Prompts

The report says they "exceeded targets in every category." What targets? Are any of these a signal that resilience actually improved?

What would a student who became more resilient do differently that you could actually observe? Name something specific.

The Dean approved budget renewal based on this report. If you were the Dean, what single question would you have asked before saying yes?

Pattern

A — Metric Obsession

This is a clean example of Goodhart's Law. The Wellness Center set attendance, reach, and page view targets — all of which are outputs of their activity, not signals of the outcome they said they cared about. "Resilience" is never operationalized. Nobody asked what a more resilient student would do differently.

Satisfaction ratings are particularly misleading here. A 94% satisfaction rate tells you students found the workshops pleasant. It tells you nothing about whether they handled their next setback differently than they would have without attending.

A better evidence plan would start with: "What would a resilient student do that a less resilient student wouldn't?" — e.g., seeks help earlier, doesn't withdraw from classes after a bad grade, reconnects with advisor after a setback. Those behavioral signals can be observed without a new survey instrument.

Group Notes

Discuss Round 1 prompts before continuing

New Information — Mid-Year Update

Six months after the budget renewal, the Dean receives a separate report from the Registrar. Voluntary withdrawal rates among first-year students increased by 11% compared to the prior year. The counseling center reports a 40% increase in crisis intervention appointments — students arriving in acute distress rather than for preventive support.

A student focus group conducted by the Dean of Students found that most students who attended the Resilience workshops reported they were "interesting" but said they hadn't changed how they handled stressful situations. Several noted they felt they should be more resilient after attending, which made them more reluctant to seek help when they were struggling — not less.

Discussion Prompts — Round 2

The data suggested the initiative was working. The system told a different story. Which piece of evidence was the real signal — and why wasn't it in the original plan?

The focus group surfaced something unexpected: the workshops may have created a new problem. Students felt they should be resilient and became less likely to ask for help. What pattern does this introduce?

What would you change in the evidence plan — specifically, what would you have looked for at the 60-day mark that might have caught this early?

Group Notes — Round 2

Their Goal

Intervene earlier when students are academically at-risk, reducing the number of students who reach academic probation or withdraw before completing their first year.

Their Theory of Change

Students fall through the cracks because faculty don't flag struggling students early enough, and advising staff are reactive. A structured alert system will create timely contact and prevent escalation.

Their Plan

Deploy an early alert platform. Train faculty to submit alerts when students miss class or perform poorly on early assessments. Configure system to auto-assign advisors to flagged students within 48 hours. Hold monthly dashboard reviews with advising director.

Their Evidence Plan

Track number of alerts submitted by faculty. Track percentage of flagged students contacted within 48 hours. Track number of advising appointments generated from alerts. Report monthly to the Provost.

Year-End Report — What They Said

"Faculty submitted 1,847 alerts. 91% of flagged students were contacted within 48 hours. Advising appointments increased by 34%. The Early Alert System has created a more proactive advising culture across the institution." The Provost highlighted this in the board report as a model for student success infrastructure.

Group Diagnosis — Round 1

Discussion Prompts

The goal was to reduce academic probation and first-year withdrawal. Which of the four metrics in their evidence plan is most directly connected to that goal — and which have nothing to do with it?

"91% of flagged students were contacted within 48 hours." Is being contacted the same as being helped? What would a signal of actual intervention effectiveness look like?

The Provost called this a "model for student success infrastructure." What is the difference between infrastructure for success and success itself?

Pattern

A — Metric Obsession (with elements of C)

Primary pattern is A — all four metrics measure the alert system's activity, not the outcome the goal describes. Alerts submitted, contacts made, and appointments generated are all things the system does. None of them tell you whether students who were flagged avoided probation or persisted to year two.

There's a C element too: the leverage point was correctly identified (early intervention) but the evidence system measured the mechanism (contacts) rather than the outcome the mechanism was supposed to produce. It's possible to perfectly execute the alert process while the underlying problem — students not getting substantively helped — continues unchanged.

A signal of the goal actually moving: "The rate of academic probation among students who received an alert is lower than among students who would have qualified for an alert but weren't reached." That requires looking at the counterfactual — which nobody built into the plan.

Group Notes

Discuss Round 1 prompts before continuing

New Information — Retention Data

The Office of Institutional Research pulls first-year retention data. Overall first-year retention is unchanged from the prior two years. Academic probation rates are statistically flat. When OIR filters specifically for students who received alerts, 68% of them still ended the year on probation or withdrew — the same rate as flagged students in the year before the system was deployed.

An informal survey of advising staff reveals a second problem: faculty began submitting alerts for minor infractions — a single missed class, one late assignment — to avoid being seen as not using the system. Advisors report spending 60–70% of their alert-response time on students who turned out not to need intervention, leaving less capacity for students in genuine difficulty who weren't flagged at all. One advisor noted: "We're very busy, and nothing is getting better."

Discussion Prompts — Round 2

Faculty began over-alerting to comply with the system, which consumed advisor capacity. Does this change your diagnosis? What pattern does this introduce?

The original leverage point — earlier contact — was reasonable. What was the actual binding constraint that the system didn't touch?

The Provost cited this in a board report. If you had to send one follow-up question before the next board cycle, what would it be?

Group Notes — Round 2

Their Goal

Reduce the retention gap between Pell-eligible and non-Pell students by removing financial and informational barriers that cause Pell students to leave before completing their degree.

Their Theory of Change

Pell students leave because they don't fully understand their aid package, miss renewal deadlines, and are unaware of emergency funds and supplemental resources. Better information and easier access will reduce dropout.

Their Plan

Launch "Access Bridge": a dedicated financial aid counselor for Pell students, proactive outreach at key deadlines (FAFSA renewal, SAP review), a peer mentor program staffed by upper-class Pell students, and a streamlined emergency fund application — reduced from 14 fields to 4.

Their Evidence Plan

Track FAFSA renewal rates among Pell students. Track number of Pell students who meet with dedicated counselor. Track peer mentor program enrollment. Track emergency fund utilization rate. Report semester outcomes to VP of Enrollment.

End-of-Year Report — What They Said

"FAFSA renewal rates among Pell students rose from 71% to 89%. Emergency fund utilization tripled. 94 peer mentor pairings were made. Pell students reported high satisfaction with the dedicated counselor." The VP highlighted the FAFSA renewal rate improvement at the trustees meeting as a sign of meaningful progress.

Group Diagnosis — Round 1

Discussion Prompts

This plan is better designed than the previous two — the theory of change is stated, the interventions are specific, the evidence plan connects to the activities. What, if anything, is still missing?

FAFSA renewal rising from 71% to 89% is a real, meaningful signal. But what is it a signal of, exactly? Is it the same thing as reducing the retention gap?

The theory of change says Pell students leave because of information and access barriers. Is that the only reason — or the main reason — Pell students leave? What might be missing from the diagnosis?

Pattern

C — Wrong Leverage Point (partially)

This is deliberately the most ambiguous scenario. The plan is genuinely good — the theory of change is explicit, the interventions are specific, the evidence measures connect to the activities. This is not a bad plan. It may be an incomplete one.

The leverage point question is whether the root cause of the Pell retention gap is primarily informational and procedural — or whether it runs deeper into financial precarity, belonging, work obligations, food and housing insecurity. If the gap is driven by things the Financial Aid office can't touch (hours worked, housing instability, family obligations), then improving FAFSA renewal rates and emergency fund access may be real and meaningful without being sufficient to close the gap.

The FAFSA renewal rate is a legitimate leading indicator — but it's a signal of procedural access, not persistence. A better evidence plan would include a one-year retention comparison between Pell students who engaged with the program and those who didn't — even a small cohort study. The goal was reducing the retention gap, and that number doesn't appear anywhere in the evidence plan.

Group Notes

Discuss Round 1 prompts before continuing

New Information — Two-Year Retention Data

Two years into the program, OIR pulls longitudinal retention data. First-year retention among Pell students improved by 4 percentage points — a statistically meaningful gain. The retention gap between Pell and non-Pell students narrowed slightly but did not close. The students who showed the largest retention gains were those who met with the dedicated counselor and completed FAFSA renewal on time — consistent with the program's theory of change.

However, a deeper analysis reveals that students who left despite engaging fully with Access Bridge shared a common profile: they were working more than 25 hours per week off-campus, and most cited financial stress unrelated to tuition — housing costs, supporting family members, transportation — as the primary reason for leaving. The emergency fund, capped at $500 per year, covered tuition-adjacent needs but not the structural financial pressures that drove departure for this subgroup.

Discussion Prompts — Round 2

The program worked — partially. For students it could reach, it made a difference. For the students it couldn't reach, the leverage point was genuinely wrong. What does this tell you about how to scope a goal?

The emergency fund cap of $500 addressed the symptom (financial need) but not the structural driver (housing, family obligations, work hours). Is this a design flaw, or a resource constraint? Does the distinction matter for assessment?

Looking across all three scenarios today: which one was hardest to diagnose, and why? What made the Access Bridge scenario different from Scenarios 1 and 2?

Group Notes — Round 2 & Final Reflection

Session Complete