Quality Of Evidence

The podcast discusses the critical importance of qualitative user research in product development, emphasizing that indirect feedback sources like analytics, support tickets, and sales calls provide limited insight. While these signals can highlight issues, they often lack context and risk misinterpretation due to team assumptions. To make informed decisions, product teams should prioritize direct user research methods such as interviews and usability testing, ideally using story-based questioning techniques that uncover the full context of user experiences.

A structured approach to evidence evaluation is recommended, combining quantitative data, expert opinions, and qualitative insights - referred to as the "ladder of evidence." High-quality interviews that capture detailed user stories yield stronger signals than superficial feedback, though even lower-quality interviews are better than no customer interaction. The discussion also explores how AI tools can support research by identifying patterns in transcripts, but stresses that human judgment remains essential. Ultimately, the focus is on designing systems that guide teams toward better research practices by providing transparent, actionable feedback and coaching to improve the quality and impact of user insights over time.

• What if you conducted a story-based user interview this week using only questions that uncover behavioral context?

  • Move: Identify one active user of your software, schedule a 20-minute call, and ask only context-driven questions: "What were you trying to do when you used [feature]?", "What happened just before you encountered this issue?", and "Did the solution actually resolve your need?"
  • Why Now? Indirect signals (e.g., feature usage drops, support tickets) are accelerating - waiting to act on assumptions risks building irrelevant features. A single well-framed interview can reveal a high-leverage bug fix or workflow improvement.
  • Expected Upside: Uncover at least one previously unknown friction point in your product flow that can be fixed in under a week, improving retention or task success rate for a user segment.

• What if you audited your last three sources of user feedback and classified each as quantitative data, expert opinion, or qualitative evidence?

  • Move: Take three recent inputs (e.g., a support ticket, an analytics report, and a sales call note), label them using Henrik Nieberg's Triangle, and note whether any decisions were made without all three forms aligning. Add a fourth column: "Assumption Risk" (Low/Medium/High).
  • Why Now? As a solo developer, your runway depends on high-confidence decisions. Misreading a weak signal (e.g., one user's complaint) as a widespread need can lead to wasted effort. This audit creates an immediate filter for decision quality.
  • Expected Upside: Avoid at least one misaligned feature build over the next month by catching a high-assumption-risk signal early, saving 10 - 20 hours of development time.

• What if you used AI to analyze a user interview transcript but manually validated every generated insight against the original audio?

  • Move: Record and transcribe a recent user conversation (or simulate one with a power user), feed it into an AI note-taker (e.g., Vistily, Otter.ai + GPT), extract 3 suggested insights, then re-listen to the recording to confirm or correct each one. Document discrepancies.
  • Why Now? AI speeds up analysis, but solo developers can't afford to act on hallucinated or shallow insights. Validating AI output builds a feedback loop that improves both your interviewing and interpretation skills.
  • Expected Upside: Develop a personal calibration standard for AI-assisted research - increasing speed without sacrificing accuracy - and identify one recurring user need that can shape your next MVP iteration.

• Conduct at least one user interview per week using a story-based format, asking specific contextual questions like "What were you trying to do?" and "Did the solution actually work?" to uncover root needs.
• Triangulate every product decision by collecting input from at least three sources: quantitative data (e.g., analytics), expert opinions (e.g., support logs), and direct qualitative evidence (e.g., recorded user interviews).
• Implement a simple rubric to score the strength of user feedback signals (e.g., low, medium, high) based on context, specificity, and observational validation, and document it for consistent team use.
• After each user interview, summarize key insights in a structured template that separates observed behaviors, stated needs, and inferred problems to reduce assumption projection.
• Use AI tools to transcribe and analyze interview notes, but schedule a weekly review session to manually validate AI-generated themes and ensure they align with actual user context.

1. Direct user research (interviews, observations) is essential. Indirect feedback from analytics, support tickets, and sales calls often lacks context, leading teams to misinterpret problems and build solutions based on assumptions.

2. Use all three sources of evidence before deciding. Henrik Nieberg's triangle requires quantitative data, expert opinions, and qualitative evidence to align - never rely on just one or two sources.

3. Behavioral analytics show what happens, not why. Without qualitative context, teams risk acting on symptoms rather than root causes, resulting in ineffective fixes.

4. Support tickets and sales calls provide symptoms, not full stories. A user reporting a bug (e.g., iPad rotation issue) may not reveal the task, goal, or whether the solution actually worked - deeper questioning is needed.

5. Story-based interviewing captures full context. Ask: what were you trying to do, when did the issue arise, what need led to the behavior, and did the solution resolve the unmet need? This prevents acting on incomplete information.

6. AI tools can analyze transcripts and identify patterns, but human interpretation remains critical. AI-generated insights still require depth and contextual understanding to be truly useful.

7. Always validate assumptions with direct user feedback. Teams often project their own interpretations onto limited data; direct interaction prevents building features based on guesses.

8. The Ladder of Evidence framework shows higher effort yields higher-value insights. Distinguish low-value signals (e.g., weak anecdotes) from meaningful evidence by demanding stronger data before making decisions.

9. Common interview pitfalls include misclassifying stakeholder interviews as customer interviews and collecting preferences without context. Avoid conducting non-usability tests while seeking usability feedback - use proper techniques.

10. Balance evidence quality with practical decision-making. Perfect data is rare; work with imperfect evidence while using a rubric to evaluate confidence and signal strength.

11. Even poorly conducted interviews provide some signal, but signal strength varies. The goal is to communicate strength transparently without discouraging teams from talking to customers at all.

12. Product tools should coach users toward better interviewing skills. Provide actionable nudges (e.g., "Next time, ask why") and explain why certain approaches are superior, using real transcript examples to highlight weak vs. strong signals.