Amazon
Software Development Engineer – AI/ML Networking Disaggregated Inference, Annapurna Labs , Elastic Collectives
On-siteFull-time
$223,600 to $223,600 a year
About the job
Every token a large language model generates depends on data reaching the right accelerator at the right moment. As AI models outgrow any single chip, the network between accelerators becomes the bottleneck that decides how fast — and how affordably — the world's largest models can serve real users. That network layer is what our team builds. We're looking for an engineer to work at the frontier of disaggregated inference: splitting LLM serving into separate prefill and decode pools and moving the model's KV cache between them at the limit of what the hardware allows. Get it right and users get answers in milliseconds; get it wrong and the fastest accelerators in the world sit idle waiting...