Sahaj Garg on Designing for Ambiguity in Human Input

The podcast episode examines the concept of ambiguity in human input, distinguishing it from noise and errors as an inherent property of unclear or multifaceted information. It highlights how humans resolve ambiguity using context, tonal cues, and prior knowledge, while machine learning models face challenges due to limited context windows, which hinder their ability to process and interpret ambiguous inputs effectively. The discussion extends to speech-to-text conversion, where unstructured spoken languagemarked by slang, filler words, and varying formalitiesrequires context-aware processing to adapt to different communication styles and user intent. Key challenges include handling background noise, accents, jargon, and the need for models to leverage contextual information to improve accuracy, especially in voice-first systems like Whisper.

The episode further explores types of ambiguity, such as polysemous words, sentence structure confusion, and stylistic variations in language depending on the audience (e.g., texting vs. professional communication). It addresses the limitations of traditional audio models and the potential of large language models (LLMs) in integrating context, history, and external prompts to enhance speech recognition. Strategies for improving model performance include contextual training with vocal metadata, data augmentation, and refining outputs through instruction tuning aligned with user preferences. The discussion also touches on balancing personalization with consistency, the role of user feedback in refining AI systems, and the importance of context compression and inference optimization in managing ambiguity and ensuring efficient, accurate AI interactions.