Amazon Bedrock Evaluations: Custom Metrics for Generative AI
Amazon Bedrock Evaluations now allows users to define custom metrics for evaluating generative AI applications, enhancing the existing LLM-as-a-judge framework. This empowers businesses to tailor evaluations to their specific needs, moving beyond pre-built metrics like correctness, completeness, and faithfulness. The platform supports both quantitative (numerical) and qualitative (categorical) scoring, enabling the creation of ordinal and nominal metrics, even adapting for classification tasks. A simplified user experience is provided through pre-built templates on the AWS Management Console, alongside the option to create custom metrics from scratch. Users can save and reuse custom metrics across multiple evaluation jobs or import them from JSON files. The system incorporates dynamic content integration using template variables like {{prompt}}, {{prediction}}, and {{context}}, seamlessly injecting data into evaluation prompts. Customizable output control allows for consistent results using a recommended schema or defining custom formats. This feature is particularly beneficial for evaluating adherence to brand voice or custom categorical rubrics. Both the Amazon Bedrock console and the Python SDK offer ways to implement custom metrics. For model evaluation, the JSON format for uploading custom metrics includes the metric name, instructions (with input variables), and an optional rating scale. The dataset format requires a ‘modelResponses’ field for BYOI (Bring Your Own Inference) jobs. RAG (Retrieval Augmented Generation) evaluation offers a similar structure but with added flexibility including ‘referenceContexts’ for comparing expected and actual retrieved passages. The article details the variables available for data injection and provides examples of using the console and Python SDK to create model and RAG evaluation jobs with custom metrics. While the system offers significant control over evaluation, it currently only supports LLM-as-a-judge and doesn’t allow for custom AWS Lambda functions or endpoints for code-based custom metric evaluators. The target audience includes businesses and developers working with generative AI applications, seeking more refined and business-aligned evaluation capabilities. The potential drawback is the current limitation to LLM-as-a-judge, potentially restricting flexibility for complex evaluation scenarios.
Amazon AI automation capabilities are enhanced through Bedrock’s evaluation framework, which provides sophisticated metrics for assessing generative model performance.
While many organizations rely on basic chatgpt automation metrics, Amazon Bedrock Evaluations offers more sophisticated custom measurement capabilities for enterprise AI applications.
