Defining What Good Looks Like

The podcast discusses the challenges of defining and measuring success in product development, particularly in the context of AI. A central theme is the importance of clearly defining "what good looks like" by establishing measurable outcomes that reflect both customer and business value. This involves moving beyond technical performance metrics - especially in AI, where outputs are non-deterministic - to focus on real-world impact, such as user satisfaction, feature adoption, and retention. The discussion emphasizes that evaluating AI should be rooted in customer feedback and behavioral data rather than generic or internally defined criteria like "helpfulness" or "conciseness," which may not capture context-specific value.

The conversation draws parallels between current AI product challenges and past technological shifts, such as the rise of mobile and user-generated content, where metrics evolved from basic performance indicators to deeper, value-based measures. It highlights the need for cross-functional collaboration to align engineering, business, and user experience perspectives, ensuring that AI features deliver meaningful outcomes. Additionally, the podcast stresses the importance of organizational clarity - defining roles, processes, and expectations - before applying AI to optimize workflows. Ultimately, the focus remains on intentionality: questioning the impact of work, testing assumptions, and avoiding the trap of automating tasks that lack inherent value, reinforcing timeless principles of clarity, critical thinking, and outcome-oriented development.

• What if you redefined success for your AI feature using only customer behavior and business outcomes?

  • Move: Identify one AI-powered feature in your product and replace its current technical metric (e.g., response time, accuracy score) with a paired customer-and-business outcome metric (e.g., % of users who complete a key task after using the feature + % of those who return within 7 days).
  • Why Now?: AI features are often judged on proxies like "helpfulness" scores that don't correlate with real-world use; aligning metrics now prevents wasted iteration on features that look good technically but don't move the needle.
  • Expected Upside: You'll detect low-value AI features early, focus development on high-impact changes, and increase user retention by tying improvements directly to observed behavior.

• What if you audited your current AI automation efforts to eliminate work that shouldn't exist?

  • Move: List all tasks your AI automates or assists with, then for each ask: "Would we create this task manually if AI didn't exist?" If not, pause or deprioritize that automation and redirect time to a high-friction user moment.
  • Why Now?: Solo developers can easily fall into the trap of automating busywork (e.g., generating boilerplate content no one reads); cutting these now frees up time before bad habits scale.
  • Expected Upside: You'll avoid compounding inefficiencies, reduce technical debt, and reallocate effort to features that solve actual user problems, improving product clarity and impact.

• What if you built your own contextual evaluation system for AI outputs instead of relying on generic benchmarks?

  • Move: For your core AI feature, define "good" by observing 10 real user sessions, then create a simple rubric based on what users actually do (e.g., "user edits the output once and sends it" = success) and apply it to every update.
  • Why Now?: Off-the-shelf evals (e.g., "coherence," "helpfulness") are too vague and mislead solo developers into optimizing for abstract qualities instead of user outcomes.
  • Expected Upside: You'll build more effective AI features faster by focusing on real usage patterns, reduce rework from misaligned expectations, and differentiate your product through hyper-relevant outputs.

• Define clear, measurable success criteria for every feature or product update by specifying both customer value (e.g., time saved, task completion rate) and business value (e.g., activation, retention impact) before development begins.
• Implement customer feedback loops early - such as in-app surveys or behavioral tracking - to evaluate AI-generated outputs based on real user perceptions of helpfulness, not just technical accuracy.
• Audit existing workflows before applying AI automation: identify high-impact, well-defined processes worth optimizing, and deprioritize automating low-value or broken workflows.
• Establish cross-functional input (even informally) when designing AI features - consult potential user needs, business goals, and technical constraints to ensure alignment and avoid building in isolation.
• Start with simple, observable metrics tied to core user actions (e.g., "% of users who complete X key task") and evolve them iteratively as the product matures, rather than adopting generic or off-the-shelf AI evaluation frameworks upfront.