Ep. 014 - Finding Miscompiles For Fun, Not Profit (AI Infrastructure) | Justin Lebar & Jordan Nanos

Description

Justin Lebar (jlebar.com) recently spent $10,000 in an afternoon, uncovering critical miscompiles across NVIDIA's PTXAS, LLVM's AMD GPU, and X86 backends. He joins Jordan Nanos (@JordanNanos) to detail his methodology, which combined traditional fuzzing techniques with novel LLM-assisted bug finding. Their discussion highlights the unique challenges of detecting flaws in less-tested ML compilers compared to mature CPU environments.Lebar shares specific high-severity X86 findings, including an atomic operation bug that splits into two non-atomic operations. They explore the comparative efficacy of fuzzing versus LLM agents in identifying these elusive errors. This episode offers critical insights into compiler security and the burgeoning role of AI in automating rigorous code verification for AI infrastructure.FULL ARTICLE00:00 Introduction and Content Overview00:25 Justin Lebar's Background and Recent Project00:59 Fuzzing Techniques for Compiler Bugs01:56 Motivation Behind the Project02:48 Challenges in Bug Detection in GPU and ML Compilers04:13 Bug Severity and Findings in AMD and x8605:38 Using LLMs to Read and Find Bugs in Code07:56 Impact of New Models and UltraCode Mode12:18 Estimating Time and Effort Without AI Assistance14:22 Limitations of Manual Code Review for Bugs15:03 Optimism About AI in Software Development16:17 Next Steps and Future Projects18:11 Key Takeaways for Developers and Researchers21:48 Call for Community Engagement and Scientific Approach