Understanding Mixes
Last updated
Last updated
Mixes are a core feature of Crosshatch that leverage multiple AI models through a single API. This section explains the types of mixes available, their benefits, and considerations for use.
Crosshatch currently offers two primary types of mixes:
These mixes automatically route requests to the current top-performing model based on industry leaderboards. As leaderboard rankings change, the mix updates to ensure access to the best-performing model.
These combine outputs from multiple models to generate comprehensive responses. By leveraging the strengths of various models, synthesis mixes aim to produce higher quality outputs for complex tasks. These mixes build on the Mixture of Agents approach pioneered by Together.ai
Task Optimization: Different models excel at different tasks. Mixes can select or combine models best suited for specific types of requests.
This approach can lead to significant performance gains. According to evaluations, the SynthCode Mix performs 18% better than the current leader in Bigcodebench Instruct Hard, a benchmark for difficult coding tasks. Specifically, it scored 31.1% (Pass@1) on 148 problems from this dataset, compared to the next best model, GPT-4.
This approach can also lead to cost savings and latency improvements, depending on the routing logic in the mix, compared to relying on the largest foundation models for all tasks.
Automatic Updates: AI model leaderboards change frequently. Mixes allow users to always access top-performing models without manual updates to their integration.
Simplified Integration: Users can access multiple models through a single API, reducing the complexity of managing multiple integrations.
While mixes offer significant benefits, they also present certain challenges:
Prompt Optimization: As the underlying models in a mix may change over time, it's challenging to highly optimize prompts for a specific model. Users should focus on creating robust, generalized prompts that work well across multiple models.
Consistency: Outputs may vary as models change. Applications should be designed to handle potential variations in responses.
Latency: Some synthesis mixes may have higher latency than single-model requests due to the processing of multiple models. While multiple models may be accessed in parallel, multiple sequential rounds of such parallelization may occur, and judgment and synthesis of those outputs typically follows afterward in sequence.
Crosshatch is continually tuning new mixes. If you have a specific use case that isn't covered by existing mixes, you can request a new mix through our Mix Request Form.
Mix Selection: Crosshatch automatically selects the most appropriate mix based on the API call parameters. Users can also specify a particular mix if desired.
Performance Monitoring: Crosshatch continuously monitors the performance of mixes and updates them as necessary to ensure optimal results.
Documentation: Each mix has specific documentation detailing its use cases, performance characteristics, and any special considerations. Users should refer to this documentation when integrating mixes into their applications.
Understanding how mixes work and their implications for application design is crucial for effectively leveraging Crosshatch's capabilities. As the AI landscape evolves, mixes provide a flexible approach to accessing state-of-the-art language models while minimizing integration complexity.
