Get in Touch
 Duration 35 hours

Course Outline

Introduction and Diagnostic Fundamentals

  • Understanding failure modes in LLM systems and Ollama-specific challenges
  • Setting up reproducible experiments and controlled testing environments
  • Mastering the debugging toolkit: local logs, request/response captures, and sandboxing

Reproducing and Isolating Failures

  • Methods for generating minimal failing examples and seeds
  • Distinguishing stateful from stateless interactions to isolate context-related issues
  • Managing determinism, randomness, and controlling non-deterministic behaviors

Behavioral Evaluation and Metrics

  • Applying quantitative metrics: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies
  • Conducting qualitative assessments: human-in-the-loop scoring and rubric design
  • Verifying task-specific fidelity and defining acceptance criteria

Automated Testing and Regression

  • Writing unit tests for prompts and components, as well as scenario and end-to-end tests
  • Establishing regression suites and golden example baselines
  • Integrating CI/CD for Ollama model updates and implementing automated validation gates

Observability and Monitoring

  • Implementing structured logging, distributed tracing, and correlation IDs
  • Tracking key operational metrics: latency, token usage, error rates, and quality signals
  • Configuring alerts, dashboards, and SLIs/SLOs for model-backed services

Advanced Root Cause Analysis

  • Tracing through complex prompt graphs, tool calls, and multi-turn interactions
  • Performing comparative A/B diagnosis and ablation studies
  • Analyzing data provenance, debugging datasets, and resolving dataset-induced errors

Safety, Robustness, and Remediation Strategies

  • Applying mitigations: filtering, grounding, retrieval augmentation, and prompt scaffolding
  • Using rollback, canary, and phased rollout patterns for model updates
  • Conducting post-mortems, capturing lessons learned, and driving continuous improvement loops

Summary and Next Steps

Requirements

  • Proven experience in developing and deploying LLM applications
  • Knowledge of Ollama workflows and model hosting practices
  • Proficiency with Python, Docker, and foundational observability tools

Target Audience

  • AI engineers
  • MLOps professionals
  • QA teams managing production LLM systems

Custom Corporate Training

Training solutions designed exclusively for businesses.

  • Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
  • Flexible Schedule: Dates and times adapted to your team's agenda.
  • Format: Online (live), In-company (at your offices), or Hybrid.
Investment

Price per private group, online live training, starting from 8000 € + VAT*

Contact us for an exact quote and to hear our latest promotions

Provisional Upcoming Courses (Contact Us For More Information)

Related Categories