K2-Chat

System architecture

Cloudflare Pages (Next.js)
         │
         ▼
Cloudflare Workers  ──►  Google Gemini API (inference + embeddings)
         │
         ├──► Amazon Cognito (auth)
         ├──► Amazon DynamoDB (sessions, messages, jobs)
         ├──► Amazon S3 (documents)
         ├──► Amazon SQS (async jobs)
         └──► Pinecone (vector search)
                  ▲
S3 ──► SQS ──► AWS Step Functions ──► AWS Glue (Apache Spark)

The frontend talks to Cloudflare Workers, which serve as the global edge API. Workers authenticate users via Amazon Cognito (including Google OAuth via Hosted UI), persist chat data in DynamoDB, and call the Google Gemini API directly for multimodal (text + image) inference and embeddings via real function-calling — the model decides for itself whether to retrieve documents or search the web, rather than always prefetching both.

For RAG, documents land in S3, trigger SQS, and are processed by a Lambda worker (small documents) or AWS Step Functions orchestrating AWS Glue Apache Spark jobs (large documents) that chunk, embed, and index vectors in Pinecone. Retrieved matches are re-ranked by a small PyTorch model (ml/reranker/, trained offline, served as a pure-TypeScript forward pass with zero production ML runtime cost) before being handed to the LLM. A scheduled weekly pipeline exports chat messages to S3 and runs a Glue analytics job over them.

The original self-hosted FastAPI/gRPC/DigitalOcean implementation is preserved in legacy/ for reference and interview deep-dives.

AI/data engineering highlights

• Agentic RAG — real Gemini function-calling (workers/src/lib/gemini.ts, runAgenticChat in workers/src/lib/models.ts) instead of a heuristic always-on-RAG/keyword-gated-search design.
• Fine-tuned reranker — a small PyTorch MLP (ml/reranker/train.py) trained on labeled query/document pairs, exported to plain-JSON weights, and re-implemented as a dependency-free TypeScript forward pass (workers/src/lib/reranker.ts): real fine-tuning with zero inference-time ML runtime cost.
• Measured RAG eval harness (workers/evals/) — retrieval precision and LLM-judge relevance scored against a fixed test set, with and without the reranker.
• Idempotent, traced, cost-optimized pipeline — exactly-once SQS processing under at-least-once delivery, requestId correlation across Workers → SQS → Lambda → Glue, and a semantic response cache in Workers KV.
• Distributed processing benchmark (logs/DISTRIBUTED_PROCESSING_BENCHMARK.md) — real measured wall-clock throughput from partition-count tuning on local Spark, plus the actual Java/Hadoop compatibility issue found and worked around along the way.

Tech stack

Directory structure

/
├── cdk/                    # AWS CDK infrastructure
├── workers/                # Cloudflare Workers API (TypeScript) + RAG evals
├── lambda/                 # Embedding worker, chat export (Python)
├── spark/                  # Apache Spark / AWS Glue jobs (PySpark)
├── ml/                     # Offline PyTorch reranker training
├── frontend/               # Next.js app
├── shared/                 # Shared types/schemas
├── tests/                  # Cross-service tests
├── scripts/                # Dev/ops helpers
├── logs/                   # Project memory / ADRs / cost guide
├── legacy/                 # Original FastAPI/gRPC/DigitalOcean code

Getting started

Prerequisites: Node.js 22+, Python 3.12+, AWS CLI configured, Cloudflare Wrangler CLI configured, AWS CDK CLI installed.

# Install all dependencies and start local dev
make dev

# Deploy AWS resources
cd cdk && cdk deploy --all

# Deploy Cloudflare Workers
cd workers && wrangler deploy

# Deploy frontend to Cloudflare Pages
cd frontend && npm run deploy

See logs/REBUILD_PLAN.md for the full implementation roadmap, logs/COST_GUIDE.md for cost guardrails, and logs/DEPLOYMENT_RUNBOOK.md for step-by-step first-time AWS + Cloudflare deployment.

Test credentials

Request tracing

Every API request gets a requestId (generated in workers/src/middleware/logger.ts), logged on every Workers log line for that request. For document uploads, the same id is threaded through to the SQS message body, the Lambda embedding worker’s structured logs, and (for large documents) the Step Functions execution name/input and Glue job arguments.

To trace a single request end to end: find the requestId in the Workers response logs, then search CloudWatch Logs Insights across the Lambda and Glue log groups for that same requestId:

fields @timestamp, @message
| filter @message like /<requestId>/
| sort @timestamp asc

Documents that go through the default S3-event-triggered Lambda path (the common case) don’t carry a Workers-issued requestId, since that trigger is native S3→SQS, not a Workers API call — the Lambda worker mints its own id in that case so its own logs are still correlated, just not to the Workers-side request id.

Cost

Designed to run within AWS, Cloudflare, and Gemini free tiers. logs/COST_GUIDE.md documents exact service-by-service costs, hidden traps, and kill switches.

License

MIT