← Field Notes

Secure AI Traffic with Agent Gateway and Kubernetes

This video introduces Agent Gateway, a Kubernetes-based proxy that routes AI traffic to different LLM providers or local models using the Gateway API. It's a powerful tool for managing AI traffic in a scalable, secure, a

4 min read

Secure AI Traffic with Agent Gateway and Kubernetes

Source: Secure your AI traffic using LLM gateways! — That DevOps Guy (https://www.youtube.com/shorts/X7PKNRP_p5I) · published 2026-06-03
Speaker(s): Not stated
Relevant to: Agentic OS, BexarByte, Cross-business

TL;DR

This video introduces Agent Gateway, a Kubernetes-based proxy that routes AI traffic to different LLM providers or local models using the Gateway API. It's a powerful tool for managing AI traffic in a scalable, secure, and flexible way, especially useful for AI-native operations and infrastructure management.

Key ideas

  • [00:07] Using Kubernetes Gateway API to route AI traffic to different LLM providers or local models.
  • [00:30] Agent Gateway is a next-gen proxy that supports routing to LLM providers, MCP servers, or agents.
  • [01:07] Agent Gateway can be installed via Helm charts and CRDs in a Kubernetes cluster.
  • [01:55] Traffic can be routed based on URL paths to different LLM backends (e.g., Anthropic, Gemini).
  • [02:21] Future plans include MCP routing and agent-to-agent routing.

Tools, services & specific callouts

Tool / service What it is How it's used in the video Use-case for my ventures
Agent Gateway A Kubernetes-based proxy for routing AI traffic to different LLM providers or local models Used to route traffic to Anthropic, Gemini, and local models using Gateway API Enables secure, scalable routing of AI traffic in Agentic OS and BexarByte, allowing dynamic switching between LLM providers or local models
Helm charts Package manager for Kubernetes Used to install Agent Gateway CRDs and control plane Useful for deploying and managing Agent Gateway in Agentic OS's infrastructure
Llama CPP A local LLM inference engine Mentioned as a possible backend for routing traffic Could be used in BexarByte or Agentic OS for local model execution
LLM Cube A tool for running models on Kubernetes Mentioned as an alternative to Llama CPP Useful for deploying models in Kubernetes environments for Agentic OS or BexarByte

How AI / automation is used

Agent Gateway uses the Kubernetes Gateway API to route AI traffic dynamically based on URL paths. It allows for seamless switching between different LLM providers (e.g., Anthropic, Gemini, OpenAI) or local models (e.g., Llama CPP). This is useful for managing AI traffic in a scalable and secure way, especially in environments where multiple LLMs are used or where traffic needs to be directed to local models for cost or latency reasons.

SEO / GEO / marketing / growth

None in this video.

Infrastructure & hardware

  • [01:02] Requires a Kubernetes cluster with Gateway API enabled.
  • [01:07] Agent Gateway is deployed using Helm charts and CRDs.
  • [01:25] YAML files are used to define Gateway listeners and backends.
  • [01:30] Gateway listener is set up on port 80.
  • [01:40] Backends are defined with secrets and model-specific configurations.

Notable quotes

"With a Gateway API enabled Kubernetes cluster, we can go ahead and install Agent Gateway using their Helm charts." — [01:02]
"This means I can route to different LLM models based on URL path." — [02:15]
"I've got a playlist available. Check the link down below." — [02:40]

Actionable takeaways

  • [Agentic OS] Explore using Agent Gateway to route AI traffic between different LLM providers or local models in a Kubernetes environment.
  • [BexarByte] Consider deploying Agent Gateway to manage AI traffic in civic tech applications, enabling dynamic switching between models.
  • [Cross-business] Use the concept of URL-based routing to manage AI traffic in any venture that uses multiple LLMs or local models.

Open questions / to verify

  • The video mentions LLM Cube and Llama CPP but does not provide URLs or versions.
  • No pricing or licensing details for Agent Gateway are mentioned in the video.

Filing metadata

  • Suggested title: Secure AI Traffic with Agent Gateway and Kubernetes
  • Primary venture: Agentic OS
  • Secondary ventures: BexarByte, Cross-business
  • Type: Tooling/How-to
  • Keyword tags: Agent Gateway, Kubernetes, LLM routing, AI traffic, LLM providers, Helm charts, CRDs, Llama CPP
  • One-line index hook: Agent Gateway enables secure, scalable routing of AI traffic to different LLM providers or local models using Kubernetes Gateway API.

AI-assisted summary of a YouTube video — source. Generated by a local model and human-reviewed; verify specifics against the original before relying on them.