Tag: infrastructure
All the articles with the tag "infrastructure".
-
Improving tail latency in parallel LLM calls
Varun SrinivasanParallel LLM calls made our document search faster on average and much slower at the tail. One straggler set the latency for the entire phase.
-
Why Postgres ignored our full-text search index
Varun SrinivasanOur full-text search index existed and Postgres maintained it correctly. The shape of our query made the planner ignore it.
-
A simple fix for LLM tail latency
Zhixuan LaiSend every request twice and take the faster response.