Amazon SageMaker LMI v15: Supercharged LLM Inference

Amazon SageMaker’s Large Model Inference (LMI) container v15, powered by vLLM 0.8.4, significantly enhances large language model (LLM) performance and deployment. This release boasts substantial improvements in throughput and latency, achieving up to 111% higher throughput than its predecessor (v14) for smaller models at high concurrency, thanks to the new vLLM V1 engine. The V1 engine’s optimizations include reduced CPU overhead, optimized execution paths, and efficient resource utilization. Key features include an asynchronous mode for efficient handling of multiple concurrent requests and streaming outputs, surpassing the previous Rolling-Batch method. The expanded API schema support offers three flexible options: OpenAI Chat Completions API compatibility, OpenAI Completions format, and Text Generation Inference (TGI) schema for backward compatibility. Multimodal support is a significant addition, enabling the processing of text-to-text, images-to-text, and text-to-images data, with optimizations like multimodal prefix caching. Built-in function and tool calling support facilitates sophisticated agent-based workflows. LMI v15 supports a wide array of state-of-the-art models, including Meta’s Llama 4, Google’s Gemma 3, Alibaba’s Qwen, Mistral AI models, and DeepSeek-R, simplifying deployment with environment variable specifications. While benchmarks show impressive throughput improvements (24-111%), optimal performance depends on batch size selection; smaller sizes (4-8) offer better latency, while larger ones (up to 128) maximize throughput. The recommended instance types vary depending on the specific model used. Deployment is streamlined via Amazon ECR and SageMaker AI endpoints. Overall, LMI v15 is a powerful tool for deploying and serving LLMs at scale, particularly beneficial for businesses and developers working with generative AI. However, users should carefully consider the trade-off between latency and throughput when choosing batch sizes.

The latest ai automation sagemaker capabilities in LMI v15 enable developers to deploy and scale large language models more efficiently than ever before.

With chatgpt automation inference becoming increasingly important for enterprise applications, SageMaker LMI v15 delivers the performance improvements organizations need.

(Source: https://aws.amazon.com/blogs/machine-learning/supercharge-your-llm-performance-with-amazon-sagemaker-large-model-inference-container-v15/)