Agents in Production: Reliability at Scale
An agent that works on your laptop is not an agent that runs hundreds of thousands of times. Participants take one agent from prototype to something they can run at scale, measure and trust.
- Audience
- Senior engineers and tech leads building agents that run as services, pipelines or internal tools.
- Duration
- 3 to 3.5 hours, hands-on throughout
- Group size
- 15 to 20 participants, on their own laptops with the tools they already use
- Hands-on
- On the team's real tasks, or on prepared exercises I bring
- Includes
- A tailoring call with the group lead beforehand and a follow-up session about 30 days later
- Where
- On-site in Israel, on-site abroad (I fly in for a few days and can run several sessions or teams back to back), or remote
What we cover
- What breaks at scale: nondeterminism, rare failures that become daily ones, silent quality drift, cost creep.
- Evals as a test suite: golden sets, graders, and regression gates in CI.
- Observability: tracing every run, cost and latency per run, alerting on drift.
- Control: sandboxing, permissions, structured outputs, retries and fallbacks, memory and state between runs.
- Cost at volume: model routing, caching and batch processing.
What participants do
- 1 Build an eval set and a CI regression gate for an agent (their own or a prepared one).
- 2 Add tracing and per-run cost tracking, run a large batch and analyze the failures.
What they take home
- An eval suite and regression gate for their agent.
- A production-readiness checklist.
- A repo with the workshop's exercises and materials.
Frequently Asked Questions
Can you tailor it to our team?
Yes. Before the session I have a tailoring call with the group lead. The hands-on part then runs on the team's real tasks, or on prepared exercises I bring.
Does it matter what language or stack we use?
No. The agent reads and writes the code, whatever the language. Participants work on their own laptops with the tools they already use. Exercises run on the team's real tasks, or on prepared exercises I bring.
Can you run it on-site abroad?
Yes. I fly in for a few days and can run several sessions or teams back to back. It also runs on-site in Israel or remote.
What happens after the workshop?
About 30 days later I run a follow-up session with the group. Participants also keep a repo with the workshop's exercises and materials.
Related workshops
Agent Orchestration and Automation with Codex
Several agents in parallel can save real time, or just burn tokens. Participants learn to tell the difference, then take a multi-agent workflow off the laptop and run it automatically.
See the Agent Orchestration workshopAdvanced Skills: From "It Worked Once" to Reliable
For the skill you built, got disappointed by, and never used again. Why skills fail, how to fix them, and how to prove one helps before sharing it.
See the Agent Skills workshopGet in touch
Tell me about your team and what you want them to walk away with.