Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Trade-offs between Hosted APIs and self-hosting
- Patterns for on-premises, edge, and air-gapped environments
- Single model versus portfolio management
- Cost-per-decision analysis
Hardware Sizing and Capacity Planning
- Memory usage relative to parameter count
- CPU-only versus GPU-accelerated processing
- Planning for concurrency and throughput
- Allowing headroom for increasing task lengths
Serving Infrastructure
- Evaluating vLLM and SGLang options
- Execution via llama.cpp and GGUF
- Using Ollama for local development
- Selecting tools based on the shape of the decision workload
Quantisation While Preserving Calibration
- Precision formats and resulting memory savings
- Measuring quality degradation and calibration loss
- Per-tensor and importance-aware quantisation variants
- Running acceptance tests post-quantisation
Latency Optimisation
- Distinguishing median versus tail latency
- Impact of context length on latency
- Implementing warm starts and model caching
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPU
- Deployment on Apple Silicon and embedded systems
- Offline operation and update strategies
- Distributing weights in air-gapped settings
Containerisation and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and routing strategies
- Setting resource limits and sharing GPUs
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Logging decision outcomes and escalations
- Setting alerts for calibration drift
Versioning and Rollout
- Versioning models and adapters
- Shadow and canary deployment methods
- Pinning model files for audit purposes
- Establishing rollback procedures
Security and Multi-Tenancy
- Implementing authentication and rate limiting
- Ensuring tenant isolation for prompts and logs
- Managing network isolation and egress controls
- Maintaining audit logs of decisions
Requirements
- Experience with Linux administration and containers.
- Knowledge of model inference servers and GPU management.
- Foundational understanding of REST APIs, load balancing, and monitoring.
Target Audience
- MLOps and platform engineers managing inference infrastructure.
- SRE teams supporting AI services.
- AI engineers deploying decision models on-premises.
- Architects designing air-gapped or regulated AI deployments.
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from 3200 € + VAT*
Contact us for an exact quote and to hear our latest promotions
