Production agent systems are distributed systems: dozens of model and tool calls, each with variable latency, a price,
and a chance of failing. AgentFlow lets you design the workflow as a graph and measures how it behaves as a system
with a discrete-event simulation — before spending a cent on API calls.
Building blocks
- LLM call — model tier, prompt and completion tokens. Latency = time-to-first-token + tokens ÷ throughput (log-normal variability); cost = tokens × price.
- Tool call — log-normal latency and a failure probability (search APIs, databases, test runners…).
- Human step — review or approval with long, heavy-tailed waiting times.
- Parallel map — fans a sub-pipeline out over N items with a concurrency limit and a minimum success fraction.
- Reflection loop — check → (fail) fix → check …; pass probability grows with each iteration, up to a maximum.
- Router — sends each run down one branch with the given probabilities; joins wait only for active branches.
Execution semantics
A node starts when all of its active predecessors have finished (fork/join). Events are processed in time order from a
binary-heap event queue. A global semaphore models the provider's concurrency limit — excess calls queue.
Reliability policy
wait_k = base · 2^k · U(0.5, 1.5) (exponential backoff with jitter)
- Failed or timed-out calls are retried; timed-out requests are still billed for their input tokens.
- After the last retry an LLM call can fall back once to the small model.
- A response cache short-circuits a fraction of LLM calls.
Analytics
Thousands of seeded runs give the latency distribution (p50 / p95 / p99), cost per run and success rate. For each successful
run the critical path is traced back from the end node through the predecessor that finished last; each node's
share of the end-to-end latency shows where optimisation pays off. Pin a baseline to compare designs A/B.
Things to try
- Set retries to 0 in the research agent: success drops from ~100% to about two-thirds.
- Lower the timeout to 30 s — long generations time out, get retried and fall back.
- In the debate preset set concurrency to 1 and watch the parallel agents serialise.
- In support triage, most of the p95 latency is the human branch — pin, then lower its share on the router.
Model tiers and prices are illustrative, not quotes from any provider.