Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Addressing the risks associated with cloud LLMs, including data retention, input usage for training, and jurisdictional concerns.
- Exploring the Ollama architecture, comprising the model server, registry, and OpenAI-compatible API.
- Comparing Ollama with alternative solutions like vLLM, llama.cpp, and Text Generation Inference.
- Reviewing licensing terms for models such as Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Installing Ollama on Linux with support for CUDA and ROCm ecosystems.
- Implementing CPU-only fallbacks and optimizing for AVX/AVX2 capabilities.
- Deploying via Docker with appropriate persistent volume mapping.
- Configuring multi-GPU setups and establishing effective VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry using commands like 'ollama pull llama3'.
- Importing GGUF models from sources such as HuggingFace and TheBloke.
- Evaluating quantization levels (e.g., Q4_K_M, Q5_K_M, Q8_0) and their respective trade-offs.
- Managing model switching and understanding concurrent loading limits.
Custom Modelfiles
- Mastering Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning critical parameters such as temperature, top_p, and repeat_penalty.
- Engineering system prompts to enforce specific role-based behaviors.
- Creating, publishing, and managing custom models within a local registry.
API Integration
- Utilizing the OpenAI-compatible '/v1/chat/completions' endpoint.
- Implementing streaming responses and enforcing JSON output modes.
- Integrating with frameworks like LangChain, LlamaIndex, and custom applications.
- Configuring authentication and rate limiting via reverse proxies.
Performance Optimization
- Managing context window sizes and KV cache efficiency.
- Optimizing batch inference and handling parallel requests.
- Allocating CPU threads and ensuring NUMA awareness.
- Monitoring GPU utilization and memory pressure metrics.
Security and Compliance
- Isolating model serving endpoints for network security.
- Implementing input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and completions.
- Verifying model provenance and hash integrity.
Requirements
- Proficiency in intermediate Linux and container administration.
- A solid high-level understanding of machine learning concepts and transformer architectures.
- Experience working with REST APIs and JSON data structures.
Target Audience
- AI engineers and developers seeking to migrate from cloud LLM APIs.
- Organizations constrained by data privacy regulations that preclude cloud model usage.
- Government and defense sectors requiring air-gapped, isolated language model deployments.
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from 3200 € + VAT*
Contact us for an exact quote and to hear our latest promotions
