I built K2-Chat because I needed a private way to run multimodal LLMs. Handling text, audio, and images without sending data to third-party APIs sounded straightforward. Six months of wrestling with gRPC, debugging distributed tracing, and chasing memory leaks taught me that building production AI infrastructure differs entirely from research prototypes.
Where the Simple Approach Breaks
The tutorials all show the same pattern: frontend talks to API, API talks to model. That works for text-only chatbots. Try adding audio transcription through Whisper, vision analysis for uploaded images, plus streaming text generation, all in one request. Now persist that conversation while handling concurrent users. The naive approach collapses under that load.
- Processing an audio file through Whisper for transcription
- Running vision analysis on uploaded images
- Streaming text tokens from a language model
- Persisting the entire conversation to a database
Doing all of this synchronously in a monolithic API would be a recipe for timeouts and frustrated users. Instead, I split the system into three distinct services communicating over gRPC.
Why gRPC Over REST?
I chose gRPC with Protocol Buffers for service-to-service calls. Coming from REST APIs, the learning curve felt steep. But the numbers convinced me:
- Binary serialization: Protobuf messages are roughly 60% smaller than equivalent JSON
- HTTP/2 multiplexing: Multiple requests share a single connection without head-of-line blocking
- Strict contracts: Generated types kept the Python services and the gateway in sync as the API evolved
Multimodal Inference on a Budget
Running SmolLM, Gemma, and Whisper together on a 4GB EC2 node meant lazy-loading models on demand and aggressively caching HuggingFace weights. Cold starts dropped by 98% once models stayed warm between requests instead of reloading from disk on every call.
Observability From Day One
OpenTelemetry and Jaeger traces across every request path made it possible to find thread-blocking bottlenecks that would otherwise hide behind average latency numbers. Flame graphs turned “it feels slow sometimes” into a specific line of code blocking the event loop during streaming responses.
Security and Multi-Tenancy
Row Level Security in Postgres (via Supabase) enforces tenant isolation at the database layer rather than trusting application code to filter every query correctly. Combined with SSR cookie-based sessions and OAuth 2.0, this closed off an entire class of authorization bugs before they could happen.
What I’d Do Differently
Given the choice again, I would introduce the gRPC service boundaries earlier rather than starting monolithic and splitting later — the refactor cost more time than designing for it up front would have.