Hitesh Jain

I work on AI inference — the infrastructure that makes large models fast and affordable to run: inference engines, serving architecture, and the measurement discipline that tells you whether any of it actually got better.

Before that I spent 13 years at Meta. I started on the Ads and Pages insights platform, then built real-time infrastructure — the gateway layer that kept every Facebook device connected, and the presence system behind the green dot, handling over a billion RPC requests a minute. Later I worked on privacy engineering, and finished as a Principal Engineer in Generative AI, fine-tuning Llama through SFT, DPO and RLHF to make assistants more personal and more useful.

After Meta I spent a year on embedding models and low-latency semantic search: continual pretraining and contrastive training of a ModernBERT backbone for the commerce domain, and the serving stack to put it in front of real traffic at a P95 of about 18 ms end to end, on a single A10. The two pieces below came out of that work.

What came next grew out of the tokenmaxxing days, when Divy Vasal — a friend from college — and I were pointing coding agents at everything and watching the bills climb. I wanted to know where the money was actually going, so I parsed a month of my own Claude Code logs: 181 sessions, 25,564 model calls. About 29M unique tokens were billed as 4.35B, because every turn re-sends the whole context. You don't pay to generate, you pay to re-read. The hidden reasoning I never see was 84% of what the model wrote and roughly 60% of what it re-read.

We also tested what agents could do once we stopped rationing them. A finance agent harness — specialist agents over SEC filings, news, stocks and options, with the industry's conventions encoded as skills — reached 82.6% on Vals AI Finance Agent v2 and 89.3% on FinanceBench, at $0.13 a question. After that, four agents ran a fine-tuning sweep by themselves: 108 LoRA experiments over a long weekend on an open 20B model, HotpotQA exact match from 0.13 to 0.72. They proposed each run, scored it, and chose the next one. I read the log and broke ties.

Both worked. Neither was something you could leave running — the agents cost around $1.1K a day, per engineer. We started Coral Bricks in June 2026, on that thesis. Agent workloads are not ordinary API traffic: hundreds of turns, context that grows with every one of them, far more read than written. Serve that on open models at high throughput and stop charging for the same context over and over, and the loops that are too expensive to leave on become ordinary ones. Frontier intelligence should be cheap enough that everybody gets to leave the loop running.

I sponsor research advancing open models at leading universities, and back a handful of early startups building on open intelligence.