Ep. 002 - InferenceX 2.0 Release (Technical Staff) | Cam Quilici, Bryan Shan, Doug O'Laughlin, Jordan Nanos

Description

InferenceX, formerly InferenceMAX: https://inferencex.com/ Article (latest): https://newsletter.semianalysis.com/p/inferencex-v2-nvidia-blackwell-vs GitHub https://github.com/SemiAnalysisAI/InferenceX Article (original) https://newsletter.semianalysis.com/p/inferencemax-open-source-inference 00:00 Introduction to InferenceX 02:52 Evolution from InferenceMAX to InferenceX 06:06 Benchmarking and Performance Insights 08:43 The Scale of Benchmarking Work 11:39 Collaboration with AMD and Nvidia 14:52 The Evolution of Inference Benchmarking 17:34 Optimizations and Their Impact 20:47 Challenges in Composability 23:51 Multi-Token Prediction Explained 26:52 Cost Implications of Optimizations 31:06 Understanding Inference Workloads and Benchmarks 33:44 Future Plans for Inference Optimization 37:16 Roadmap for New Models and Data Sets 39:03 Challenges in Benchmarking Multi-Turn and Multi-Modal Data 42:44 Experiences with AI Models and Their Limitations 48:43 Skepticism About Future AI Improvements