Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 21 hours
Course Outline
Introduction to Scaling Ollama
- Overview of Ollama’s architecture and scaling factors
- Typical bottlenecks in multi-user deployments
- Best practices for ensuring infrastructure readiness
Resource Allocation and GPU Optimization
- Strategies for efficient CPU and GPU utilization
- Key considerations for memory and bandwidth
- Resource constraints at the container level
Deployment with Containers and Kubernetes
- Containerizing Ollama using Docker
- Executing Ollama within Kubernetes clusters
- Load balancing and service discovery mechanisms
Autoscaling and Batching
- Developing autoscaling policies specific to Ollama
- Batch inference methods to enhance throughput
- Balancing latency and throughput trade-offs
Latency Optimization
- Analyzing inference performance
- Implementing caching strategies and model warm-up
- Minimizing I/O and communication overhead
Monitoring and Observability
- Integrating Prometheus for metric collection
- Creating dashboards using Grafana
- Alerting and incident response for Ollama infrastructure
Cost Management and Scaling Strategies
- Cost-aware GPU allocation methods
- Considerations for cloud versus on-prem deployment
- Strategies for sustainable scaling
Summary and Next Steps
Requirements
- Practical experience in Linux system administration
- Comprehension of containerization and orchestration principles
- Knowledge of machine learning model deployment
Target Audience
- DevOps engineers
- ML infrastructure teams
- Site reliability engineers
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from 4800 € + VAT*
Contact us for an exact quote and to hear our latest promotions