AgentGrid has to treat failures as normal operational events. Workflow execution will hit tool errors, bad inputs, rejected approvals, worker failures, timeouts, retries, cancellations, and partial progress. The product should be designed for that reality from the beginning.
The most important reliability decision is to separate state. A managed worker completing successfully does not always mean the workflow completed. It may mean the worker successfully paused at an approval gate. Those are different outcomes.
The system should distinguish worker job status, workflow run status, step run status, and approval status. Collapsing those into one state creates confusing UI and dangerous automation semantics.
A workflow run might be created, queued, running, waiting for approval, completed, failed, or canceled. A step run might be not started, queued, running, completed, waiting for approval, failed, canceled, or rejected. An approval might be pending, approved, rejected, or redone.
Each state transition should be persisted. No critical workflow state should live only in memory. Definitions, runs, step runs, execution jobs, approval records, decision history, logs, output references, retry relationships, and schedule metadata all need durable records.
This persistence makes pause and resume possible. When a worker reaches an approval gate, it should persist the pause, create an approval record, and stop active execution. When approved, the system should create a continuation job and resume from the persisted state.
The same model supports retry and cancellation. A failed run should remain visible. A retry should create a new execution job or retry-linked run. Cancellation should mark incomplete work clearly instead of leaving ambiguous state.
Schedules should follow the same execution path as manual runs. The scheduler selects due workflows and creates execution jobs. It should not run workflow logic directly. Manual run, scheduled run, approval continuation, retry, and redo all become job-driven paths through the same runner.
Reliability here is not only infrastructure reliability. It is product reliability: the user can see what happened, understand what state the workflow is in, and choose the next action without guessing.