Module 4 Design
Why This Session Exists
Participants arrive with draft goals but without a coherent way to think about measurement. This session prevents two equally dangerous failures: forcing premature metrics and avoiding measurement entirely.
Session Purpose
- Establish a coherent mental model for measurement that begins with signals, not statistics
- Show how evidence evolves across the planning cycle rather than being fixed at the start
- Help participants distinguish early indicators, developing evidence, and long-term impact measures
- Connect measurement to the mid-year and end-of-year checkpoints already built into the DAP structure
- Prevent regression back to activity-based KPIs after three sessions of strong conceptual work
The Central Problem This Session Solves
After three sessions on outcome-based planning, participants typically hit a wall when asked how to measure their goals. Two opposing errors emerge:
This session introduces a third path: begin with observable signals tied to the outcome, use checkpoints to refine those signals into stronger evidence, and treat formal metrics as the output of learning rather than its precondition.
What Participants Will Walk Away With
- ✓ A practical language for talking about measurement without needing to define a metric immediately
- ✓ A completed Signal Map for at least one of their draft goals
- ✓ A clear understanding of how mid-year and end-of-year checkpoints serve different measurement purposes
- ✓ A principled way to respond to pressure for early metrics without abandoning accountability
- ✓ Alignment between their evidence plan and the DAP review cycle
Where This Sits in the Series
Module 1
Impact Thinking
What should be true? Interpreting institutional goals as observable real-world change.
Module 2
Commitment
What will we own? Translating shared outcomes into unit-level goals using OIC.
Module 3
Calibration
Are we ready? Stress-testing goals for structural integrity and appropriate scope.
Module 4 ← Now
Evidence
How will we know? Building a signal-based measurement approach that evolves over time.
Core Conceptual Framework
The Evidence Progression
Measurement is not a single moment. It is a progression from noticing to knowing, structured across the planning cycle. This framework replaces the pressure to define a perfect metric upfront with a disciplined sequence of increasingly strong evidence.
The Four Levels of Evidence
Each level is a legitimate form of evidence at different moments in the planning cycle. None is permanent — all can be strengthened over time.
01
Signals
Early Indicators
Observable signs that something is beginning to shift. Not yet validated, but worth noticing. These are the most accessible starting point.
"We notice students are asking fewer clarifying questions about the advising process." / "Fewer repeat visits to the same office."
WHEN → Available at goal-setting. Use to open the evidence conversation.
02
Patterns
Developing Evidence
Signals that recur consistently enough to suggest a real trend. Multiple sources pointing in the same direction. Appropriate for mid-year reflection.
"Student confidence ratings in advising interactions have trended upward across two consecutive semesters." / "Help-seeking is arriving earlier in the semester."
WHEN → Appropriate for mid-year checkpoint. Refine signal into trackable pattern.
03
Evidence
Validated Change
Patterns that have been confirmed across time periods, populations, or methods. Strong enough to anchor a formal finding and inform next-cycle planning.
"First-generation students now complete course planning in one interaction at a rate 30% higher than the prior year."
WHEN → Target for end-of-year reporting. This is what goes in the DAP annual review.
04
Impact
Long-Term Measures
Sustained change visible in institutional patterns over multiple years. Shared across units. Not achievable in one cycle, but the direction all unit evidence points toward.
"Four-year persistence rates among students who engaged with early advising have increased over three consecutive cohorts."
WHEN → Institutional level, multi-year horizon. Your evidence contributes to this — you don't own it alone.
The Signal Map — How to Think About Evidence Before Defining a Metric
Before asking "what will we measure," ask "what would we notice?" This sequence structures evidence thinking without requiring a metric upfront.
Question 1
If This Goal Were Moving...
What would people do differently? What would you observe in hallways, emails, conversations, or requests? These are behavioral signals — no instrument needed yet.
Question 2
Where Would We Look First?
What existing data, conversations, records, or observations already contain traces of this change? What would you check in the next 60 days without creating anything new?
Question 3
What Would We Need to Track Over Time?
If the signal is promising, what ongoing collection would let us confirm it? This is where formal measures begin to emerge — from validated signals, not from abstract planning.
Aligning Evidence to the DAP Review Cycle
The planning structure already answers the question of when to formalize measurement. Match your evidence to the checkpoint, not the other way around.
Mid-Year Checkpoint
What signals are we noticing?
- •Report on Level 1 signals observed so far
- •Note whether early signals are positive, absent, or mixed
- •Identify what would strengthen the signal into a pattern
- •Adjust activities if signals are weak or absent
End-of-Year Review
What do we know, and what do we now know to ask?
- •Present Level 2 or 3 evidence if available
- •Distinguish what changed from what you tried
- •Articulate the better question for next cycle
- •Propose any formal metric that has earned its definition
DAP Module 4 — Activity 1: Signal Mapping
Haverford College Divisional Assessment Plan Workshop | Name: _________________________ | Unit: _________________________
Module 4 — Activity 1
Signal Mapping
Translate your draft goal into observable early indicators. Complete each field below, then use the export button to print your Signal Map.
Activity 1
Building a Signal Map for Your Goal
Purpose: Move from a well-formed goal to a set of observable early indicators — without defaulting to activity counts or abstract metrics. A signal is only useful if it is directly tied to the change your goal describes. Otherwise it is noise.
A strong goal describes a direction of change within your unit's influence — not an activity, event, or program.
Q1 — What would I notice?
Observable Signals
List 3+ behavioral changes in the people you serve. Describe what they do — not what your office does. The test: could someone else observe this without you explaining your theory?
1
2
3
Q2 — Where would I look first?
Existing Sources
Name 2+ sources that already exist in your daily workflow. Resist the urge to propose a new survey. Start with what you already have access to.
1
2
Q3 — What would I track over time?
Tracking Question
Write one question you would return to at mid-year. Phrase it as a question, not a target number. This is where formal measures begin to emerge — from validated signals.
?
Which single signal from Q1 is most directly tied to your goal and most observable by someone outside your unit? Circle it above, then restate it here.
Connection Check — Before you continue, ask yourself:
If you can't check all four, revise the signal before moving to Activity 2.
Peer Review Prompts 20 min with a partner
Share your Signal Map with a partner. The reviewing partner answers these three questions out loud:
- "Could I actually observe those signals without knowing your theory of change?"
- "Are any of these really activity counts in disguise?"
- "What is the strongest signal on this list — the one most directly connected to the goal?"
DAP Module 4 — Activity 2: Evidence Timeline
Haverford College Divisional Assessment Plan Workshop | Name: _________________________ | Unit: _________________________
Module 4 — Activity 2
Evidence Timeline
Map your signals onto the DAP planning cycle. Complete the three time columns, then add your closing commitment. Use the export button to print your Evidence Timeline.
Activity 2
Planning Your Evidence Across the Cycle
Purpose: Understand that what counts as "good evidence" changes at different moments in the year. The Evidence Timeline removes the pressure to have perfect measurement now by showing that strength of evidence is expected to develop over time.
NOW → MID-YEAR
Early Signals
MID-YEAR → END OF YEAR
Developing Patterns
END OF YEAR
Validated Evidence
What signals will I watch for, and where will I look?
What would concern me if absent?
Observations · Existing records · Staff conversations
What shift or recurring pattern would tell me the goal is moving?
Is the signal strengthening?
Recurring patterns · Survey data · Comparative records
What stronger evidence could I present at year-end? Has any formal measure earned its definition?
Formal metric — only if earned
Validated change · Finding tied to observable outcome
What long-term institutional pattern is your unit contributing toward? You won't own this measure — but your evidence points toward it. Name the direction.
Small Group Review Prompts 15 min in groups of 3–4
Share one timeline. The group responds with these three questions:
- "Your early signal — is there a way to check it without creating new work?"
- "Your end-of-year evidence — does it describe a change in condition, or a change in your output?"
- "Does this timeline feel honest, or does it feel like what someone wants to hear?"
✎ Your Signal Commit
Complete this sentence before you leave today:
"Between now and mid-year, I will look for
by
Teaching Points
Misconceptions to Address Directly
These are the specific errors that will emerge in this session. Address them proactively — not defensively — as shared patterns that trip up even experienced planners. Name them before participants enact them.
Misconception 1
"We can't measure this. It's too qualitative."
This conflates measurement with quantification. Evidence does not require a number. The question is not "can we count it" but "can we observe it consistently enough to know whether it is changing?"
This conflates measurement with quantification. Evidence does not require a number. The question is not "can we count it" but "can we observe it consistently enough to know whether it is changing?"
Reframe
Qualitative evidence is evidence. A pattern noticed by three staff members across ten advising conversations is evidence. A shift in the kinds of questions students ask is evidence. Name what you observe. Consistency makes it meaningful — not quantification.
Misconception 2
"We need to define our metrics now so everyone knows what success looks like."
This assumes the metric is already known and simply needs to be stated. In most outcome-based goals, the right metric only becomes clear after observing which signals actually reflect the change you care about.
This assumes the metric is already known and simply needs to be stated. In most outcome-based goals, the right metric only becomes clear after observing which signals actually reflect the change you care about.
Reframe
A metric you define before observing any signals is a guess. A metric you define after confirming a signal across two semesters is a finding. The goal of early planning is to name the right signals, not to name the right number. The number can wait.
Misconception 3
"We already track that. Our attendance data covers this."
Attendance is an output, not an outcome. How many people showed up to your event tells you about your reach. It tells you nothing about whether those people changed in the ways your goal describes.
Attendance is an output, not an outcome. How many people showed up to your event tells you about your reach. It tells you nothing about whether those people changed in the ways your goal describes.
Reframe
Ask: "What is different about a student after this interaction than before?" If your answer is "they attended," that is not yet a signal. If your answer is "they now describe the process clearly without prompting," that is a signal. Keep going past the attendance record.
Misconception 4
"Our evidence is anecdotal. That doesn't count."
This applies a research standard to a planning context. In institutional assessment, a well-observed anecdote from a person with direct contact with students is more useful than a statistically significant finding disconnected from the outcome.
This applies a research standard to a planning context. In institutional assessment, a well-observed anecdote from a person with direct contact with students is more useful than a statistically significant finding disconnected from the outcome.
Reframe
Anecdote becomes evidence when it is noticed consistently, described specifically, and triangulated with other sources. The question is not "is this one story?" but "is this story representative of a pattern?" Start with the story. Build toward the pattern.
Pitfall — The Metric Trap
The institutional pressure to "show the numbers" pushes participants to report whatever they can count — even when the count has no logical connection to the goal. This creates a false sense of accountability.
Antidote: Ask "If this number improved, would the goal be moving? If the number declined, would the goal be failing?" If the answer to both is "not necessarily," the metric is wrong.
Pitfall — The Perfectionism Trap
Some participants refuse to name any evidence because they fear naming it incorrectly. They wait for the "right" instrument, the "right" survey, or the "right" methodology before claiming any signal.
Antidote: Imperfect signals, named honestly and held lightly, are more useful than no evidence at all. The goal is learning, not certainty. Start somewhere specific.
Pitfall — The Ownership Trap
Participants claim institutional-level outcomes as their unit evidence ("our goal is for students to graduate prepared for meaningful careers"). The evidence they name is disconnected from anything their unit actually touches.
Antidote: Return to the OIC framework. The signal should be observable within the unit's sphere of influence, not the institution's entire student lifecycle.
Pitfall — The Survey Reflex
When asked how they will know, many participants immediately propose creating a new survey. This creates new data collection burden without first asking whether the answer already exists in something they have.
Antidote: Before proposing a new instrument, ask: "Does this signal show up anywhere we already look?" Start with existing records, conversations, and observations before building new collection.
Quick Reference
Module 4 Framework Reference
A concise reference for the frameworks, distinctions, and quality checks covered in this session. The Core Framework tab has the full Evidence Progression and Signal Map detail — this tab adds the frameworks not covered there.
The Critical Bridge
From Goal to Signal
A signal is only useful if it is directly tied to the change your goal describes. Otherwise it is noise.
Goal
What change are we trying to create?
Connection Test
If this signal improved, would the goal be moving? If it declined, would the goal be failing?
Signal
What would we notice if that change were starting?
If the answer to both connection test questions is "not necessarily" — it's noise, not a signal.
Critical Distinction
Evidence vs. Metrics
Metrics should be earned, not assumed. Define metrics after signals are validated — not before.
| Metric | Evidence | |
|---|---|---|
| What it is | A predefined number or target | Any signal of meaningful change |
| When defined | Upfront, before observing | After signals confirm what to track |
| Risk if wrong | You optimize for the wrong thing | You describe something more accurately |
A metric defined before observing any signals is a guess. A metric defined after validating a signal is a finding.
Debrief Reference
Strong vs. Weak Signals — With Diagnosis
Weak Signal
Why It Fails
Strong Signal
"We will track more advising appointments."
Counts the activity, not the change in the person. More visits ≠ better navigation.
"Students arrive having already drafted questions — without prompting."
"Students will feel more confident."
Internal state with no observable consequence. You cannot see "feeling confident."
"Students resolve conflicts without requesting staff intervention."
"We will track checklist completion."
Measures compliance with a process, not change in a person. Completing a form is not learning.
"Students stop asking basic orientation questions at their first advising appointment."
"We will monitor graduation rates."
A lagging institutional indicator — too remote from unit levers to be useful as a signal.
"Course planning errors requiring staff correction decrease from prior year."
Resolving the Tension
Do We Ever Define Metrics?
01
When
After signals are validated — not at goal-setting
02
What
Stabilized evidence — confirmation, not exploration
03
How Many
Not every goal needs one — absence of a metric is not failure
You are not expected to define perfect metrics upfront. You are expected to define what you will look for.
Common Traps
Four Misconceptions — Quick Look
"We can't measure this. It's too qualitative."
Qualitative evidence is evidence. Consistency makes it meaningful — not quantification.
"We need to define metrics now so everyone knows what success looks like."
Name the right signals first. The number can wait. Metrics are the output of learning, not the input.
"Our attendance data covers this."
Attendance is an output, not an outcome. Ask: what is different about a person after this interaction?
"Our evidence is anecdotal. That doesn't count."
Anecdote becomes evidence when noticed consistently and triangulated. Start with the story — build toward the pattern.
Why It Matters
The Learning Loop
Signals
help you notice
→
Evidence
helps you understand
→
Learning
helps you decide
→
Action
is the point
If you don't use what you notice to adjust what you do, measurement has no value.