Customize Amazon Nova with SageMaker: Direct Preference Optimization
Amazon has unveiled a suite of model customization capabilities for its Amazon Nova foundation models, accessible via Amazon SageMaker AI. These capabilities allow users to adapt Nova Micro, Nova Lite, and Nova Pro throughout the model training lifecycle, encompassing pre-training, supervised fine-tuning, and alignment. A key technique highlighted is Direct Preference Optimization (DPO), an alignment method simplifying the tuning of model outputs to user preferences. DPO employs paired prompts and responses, one preferred and one not, to guide the model towards desired style and tone. Users can choose between parameter-efficient or full model DPO, balancing data volume and cost. Customized models are deployable to Amazon Bedrock for inference, with the parameter-efficient version enabling on-demand inference. The customization process leverages SageMaker training jobs and SageMaker HyperPod, offering flexibility in infrastructure selection. A streamlined workflow is presented, where users select a Nova customization recipe, submit an API request to SageMaker, and SageMaker manages the infrastructure and training process. The article details a use case involving adapting Amazon Nova Micro for structured function calling in agentic workflows, demonstrating an 81% increase in F1 score and up to 42% gains in ROUGE metrics. The process involves using DPO to align the model with human preferences, leveraging the nvidia/When2Call dataset for training. Evaluation is performed using SageMaker training jobs with evaluation recipes, providing metrics for both task-specific performance and alignment with desired behaviors. Prerequisites include quota increases for SageMaker, IAM role creation with specific policies, and cloning a GitHub repository. Finally, deployment to Amazon Bedrock is achieved using the CreateCustomModel API, enabling integration with Bedrock's native tools. While the solution offers significant benefits in customizing Nova models for specific applications, resource requirements, particularly the need for p5.48xlarge instances, should be considered.
This tutorial demonstrates how ai automation sagemaker capabilities enable developers to fine-tune Amazon Nova models using advanced preference optimization techniques.
While chatgpt automation sagemaker workflows have gained popularity, Amazon Nova offers enhanced customization capabilities through Direct Preference Optimization techniques.
