Inference on Kubernetes / Building a Multi-Avatar System with Agentic AI

For the November meetup the first preso will be from Hrittik Roy, a Platform Advocate at vCluster Labs and CNCF Ambassador, “From Local Inference to Kubernetes: What Changes When Your LLM Gets Busy?”. The presentation covers GPU scheduling, batching, KV cache and inference observability, using vLLM and NVIDIA Dynamo as examples, For the second presentation Arsen Yeremen returns to discuss agentic AI development of a multi avatar system. See the Presentation Details section below for more info. Sponsor and LA DevOps Meetup: Sponsoring this or a future meetup is great way to promote your service or product and to give back to the local DevOps Community,' Reach out to one of the hosts for details. Event Details: 6:00 - 6:30 Attendees arrive. Networking/Mingle 6:30 - 6:45 LA DevOps Welcome,Attendee Introductions 6:45 - 7:15 Hrittik Roy \- From Local Inference to Kubernetes: What Changes When Your LLM Gets Busy? 7:15 - 8:00 Arsen Yeremin - Building a Multi-Avatar System with Agentic AI 8:00 - 9:00 Networking/Mingle Presentation Details: Title: From Local Inference to Kubernetes: What Changes When Your LLM Gets Busy? Description: * Running an LLM locally is a great starting point. But what happens when one user becomes hundreds, requests compete for GPU memory, and response times start climbing? This talk follows an inference request from prompt to generated tokens, explaining prefill, decode, and the KV cache. We’ll explore how batching, model caching, and GPU scheduling affect latency and throughput, then introduce disaggregated inference: running prefill and decode on separate workers, including when the added complexity makes sense. Using vLLM and NVIDIA Dynamo as examples, we’ll connect these concepts to serving models on Kubernetes and measuring what matters: time to first token, token generation speed, and GPU utilization. Key…