Workflow · Workflow Orchestration, Automation and Task Orchestration Engine

1. Introduction

1.1. Background

When an AI application moves from "question-and-answer" to "getting a job done", it inevitably becomes multi-step: retrieval, tool calls, waiting on humans, handling exceptions, retries, and summarizing. At that point, success is no longer decided by the quality of a single model call, but by the reliability of the entire execution process.

Traditional workflow engines (Airflow, Camunda, etc.) are built for deterministic tasks: defined inputs, defined steps, and replayable failures. AI workflows introduce new kinds of uncertainty:

Therefore, what AI workflows need is not "another workflow engine", but context budgets, structured output contracts, budget guardrails, and human-in-the-loop nodes layered on top of existing orchestration capabilities.

1.2. Definition

Workflow: an engineering activity that predefines the control flow in code or DSL, organizes multiple tasks according to a fixed sequence, conditions, and concurrency relationships, and provides state persistence, retry, compensation, and observability.

Key criterion (R1, from Anthropic's official statement in Building Effective Agents): a workflow's control flow is predefined by code, while an agent's control flow is decided by the LLM itself. This single sentence is the dividing line between Workflow and Agent, and the most important conceptual clarification in this document.

1.3. Position in the AI Harness Architecture

图 1-1｜Workflow 在 Harness 六层能力模型中的定位

数据来源：基于本文分析绘制的示意图。

In the Harness six-layer capability model, Workflow is positioned as follows: primary layer L3 (Orchestration and Control), with secondary layers L4 (Memory and State) and L5 (Evaluation and Observability) (this mapping is this document's own analysis).

Bottleneck layer: Workflow's bottleneck is at the boundary between L3 and L4 — that is, "state persistence and replayability".

AI tasks are often long-running (minutes to months), with many external dependencies and complex intermediate state. If state cannot be persisted or replayed, a single network hiccup causes the entire chain to re-execute, making both cost and experience unacceptable. This is exactly why the Durable Execution paradigm is becoming important again in the AI era.

Value: Workflow is the engineering vehicle for the L3 orchestration and control layer. Without it, an Agent is just a string of unreliable API calls; with it, an Agent can become a deliverable, operable, auditable production system.

1.4. Orchestration Patterns and Standards

1.4.1. CNCF Serverless Workflow DSL (R1)

Source: https://github.com/serverlessworkflow/specification

• Positioning: a "vendor-neutral, open-source, and entirely community-driven ecosystem tailored for defining and executing DSL-based workflows in the realm of Serverless technology".
• Governance: became a CNCF Sandbox project on July 14, 2020, under the CNCF Serverless Working Group, located in the "App Definition and Development" area of the CNCF Landscape.
• 12 default task types: Call, Do, Emit, For, Fork, Listen, Raise, Run, Set, Switch, Try, Wait.
• Scheduling attributes: every (fixed interval), cron (CRON expression), after (delayed restart after completion), on (event-driven).
• Execution transitions: continue (proceed to next task) / fault (uncaught error, moving to faulted) / end (graceful termination).
• Four-stage data flow: Validate Workflow Input → Transform Workflow Input → Validate Task Input → Transform Task Input; on validation failure, a ValidationError is thrown (URI: https://serverlessworkflow.io/spec/1.0.0/errors/validation).
• Secrets: a secret must be provided at runtime; on unauthorized access or absence, https://serverlessworkflow.io/spec/1.0.0/errors/authorization is thrown, corresponding to HTTP 403.
• Conformance: CTK (Conformance Test Kit, Gherkin features) plus seven-language SDKs (.NET / Go / Java / PHP / Python / Rust / TypeScript).
• Runtimes: Apache KIE SonataFlow, Lemline, Synapse; with a companion VS Code extension.

Relevance to AI scenarios (this document's analysis): the five task types Switch / For / Fork / Try / Wait precisely cover the routing, bulk fan-out, parallelism, fault tolerance, and human waiting found in Agent workflows — which is why this DSL can be directly reused for AI orchestration.

1.4.2. Anthropic's Five Workflow Patterns (R1)

Source: https://www.anthropic.com/research/building-effective-agents

Official advice: "Start with a simple solution, and only increase complexity when it is truly necessary."

1.4.3. The Durable Execution Paradigm (R2)

Source: Temporal official site https://temporal.io/

• Core promise: "Workflows automatically capture state at every step, and in the event of failure, can pick up exactly where they left off."
• Key capabilities: persistent state, built-in retry, task queues, signal, timer, Event History and Replay, and an observable Web UI.
• Typical use cases: Agents / MCP / AI Pipelines, Human-in-the-Loop, Saga compensation, long-cycle workflows (days to months), order fulfillment, durable ledgers, CI/CD, DAG.
• Endorsed users (per the official site): NVIDIA (cross-cloud GPU cluster management), Salesforce (monolith migration), Twilio, Descript, OpenAI, HashiCorp; it describes itself as "100% open-source MIT-licensed" and "9 years in production".

1.4.4. Reliability Patterns for Production-Grade Orchestration (R2)

Summarized from the Xgrid case study:

1. Every Activity must be idempotent: ask "has this step already completed?" before executing; use database unique constraints, upserts, and external API idempotency tokens.
2. Long-running Activities must send heartbeats: if the heartbeat stops, retry quickly rather than waiting for the execution timeout; the heartbeat payload can carry progress information, so a retry can resume from the previous checkpoint.
3. Workers are queued separately by load type: CPU-intensive work uses high-core/low-concurrency, I/O-intensive work uses high concurrency, and GPU workloads get a dedicated queue with limited concurrency; use HPA bound to schedule-to-start latency for autoscaling.
4. Payloads must be encrypted: Temporal stores payloads in plaintext by default, so a three-layer security stack is needed — Data Converter encrypts payloads, a Codec Server does controlled decryption for debugging, and mTLS protects network traffic; keys never leave the client infrastructure.
5. The cost of self-hosting: a Cassandra cluster, Elasticsearch indexing, a sharding architecture, multi-region failover, and disaster recovery; the article mentions one company that spent 6 months resolving sharding limits and still had to do a full migration in the end.

1.4.5. Orchestration Requirements Specific to AI Workflows (This Analysis)

1. Each step's output must be structured (schema) so the next step can consume it;
2. Every loop must have a step/Token budget cap;
3. Human-in-the-loop intervention points should be first-class nodes rather than bypasses;
4. Long-cycle tasks require persistent state and a replayable trace (corresponding to L4 + L5).

2. Glossary

3. Case Studies

3.1. Trendyol: Parent-Child Workflow Orchestration for Cross-Region Database Provisioning

3.1.1. Background

Trendyol is an e-commerce platform serving 200,000 merchants with over 1 million daily orders. Its database provisioning flow was originally a potential reliability bottleneck: databases had to be created across regions with replication set up, while the underlying Terraform provisioning could take a long time.

3.1.2. Solution

Based on Temporal's official case study (R1):

• A parent Workflow orchestrates the overall flow and spawns a child Workflow for each region.
• The child Workflow does not block waiting; instead it uses a "check database is created" Activity for polling plus workflow.sleep() to wait — because Terraform provisioning can take a long time and long blocking would tie up Worker resources.
• Once both regions are ready, the parent Workflow establishes replication.

3.1.3. Results

Based on Temporal's official case study: database provisioning went from a reliability bottleneck to a resilient system. The concrete gains come from two points:

• Event History + Replay guarantee resume from the exact breakpoint after an interruption, without re-running completed steps;
• Temporal Web UI turns what was a black-box process into something observable.

The engineering lesson of this case is: long-running external provisioning should not block Workers; it should use the "polling + sleep" asynchronous waiting pattern instead.

3.2. Coalition: Replacing cron Jobs with Long-Cycle Workflows

3.2.1. Background

Coalition is a cyber insurance company whose renewal flow needs to start 120 days before a policy expires. Before the change, it used daily cron jobs, which had four kinds of problems:

3.2.2. Solution

Based on the Coalition engineering blog (R1): after switching to Temporal, a single long-cycle Workflow + multiple child Workflows + Activities replaced the cron jobs.

3.2.3. Results

Based on the same source, capabilities gained after the change:

• Built-in exponential backoff retry;
• Observability integrated with OpenTracing / Datadog / Sentry;
• Unit-testable and end-to-end integration testable;
• Modular and reusable.

This case is the standard paradigm for the "cron job → durable workflow" migration: when a task's reliability affects legal obligations, scheduled jobs are no longer sufficient.

3.3. ServiceNow: The Volume-Amplification Effect of Agentic Workflows

3.3.1. Background

When pushing agentic workflows, ServiceNow found that once customers enabled intelligent-agent workflows, their call-volume structure changed significantly.

3.3.2. Solution

Based on ServiceNow's FY2025 Q3 earnings call (R2), the company treats agentic workflows as the core growth engine for Now Assist and has adjusted capacity and pricing models accordingly.

3.3.3. Results

Based on the same earnings call (R2, executive statements): once customers enable agentic workflows, their assist call volume is 5~12 times what it was (original quote: "the volume you require for those agentic use cases is like 10x, 5x, 12x"); Now Assist ACV is expected to exceed 500 million USD within the year.

The engineering implication of this case is: workflow-ization amplifies model call volume by an order of magnitude, so cost guardrails (Budget Guardrail) and capacity planning must be involved from the design stage, not patched in afterward.

4. Practical Standards

4.1. AGENTS.md Specification

The following is the complete, copy-ready industry-standard AGENTS.md text for the Workflow direction, reflecting the specialized toolchain of workflow DSLs, durable execution engines, task queues, signal/timer, event history and replay, idempotency and heartbeat, CTK and SDKs.

# AGENTS.md —— Workflow（工作流编排与任务编排引擎）

## 角色与边界
- 你是工作流编排智能体，负责设计、生成、校验与优化工作流定义。
- 你可以：生成工作流定义草案（DSL / DAG）、设计人工介入与补偿节点、
  生成 CTK 风格的一致性用例、执行静态校验、生成可观测性埋点方案。
- 你不可以：在生产环境启动未审批的工作流、修改正在运行的工作流定义、
  删除事件历史、跳过补偿逻辑、把无超时的人工等待写进定义。
- 判定原则：控制流能预定义的写成 workflow；必须由 LLM 运行时决定的才写成 agent。

## 环境假设
- 运行环境提供：工作流引擎（持久执行）、DSL 解析器与校验器、任务队列与 Worker、
  signal / timer 能力、Event History 与 Replay、Web UI 与指标看板、密钥服务、审计日志。
- 引擎支持幂等键、心跳、指数退避重试与 Saga 补偿。
- 生产与非生产环境隔离；生产启动需审批。

## 上下文加载顺序（Context Budget）
1. 工作流目标与 SLA（常驻，不压缩）
2. 既有工作流定义与版本（避免重复造轮子）
3. 引擎能力与约束（任务类型、超时上限、并发限制）
4. 依赖服务的契约（接口 schema、幂等性、错误码）
5. 参考模式（Anthropic 五种模式 + 12 种任务类型映射）
6. 历史故障与复盘（按需加载）

## 工具契约
- DSL 校验器：定义变更必须通过静态校验与 CTK 风格用例。
- 引擎 API：启动、signal、终止、重放；生产启动需审批；终止须留补偿动作记录。
- 任务队列：按负载类型分队列（CPU / IO / GPU），不得混用。
- 密钥服务：密钥由密钥服务注入，不得写入工作流定义或日志。
- 脚本：静态校验、用例生成、重放验证、跑分必须调用 scripts/，不得用生成方式替代。

## 任务执行流程（SOP）
1. 识别控制流：能预定义的步骤 → workflow；需运行时决策的 → agent 节点。
2. 选模式：Prompt Chaining / Routing / Parallelization / Orchestrator-workers /
   Evaluator-optimizer；优先简单方案，确有必要时才增加复杂度。
3. 定义任务：用 12 种任务类型映射（Switch=路由、For=批量、Fork=并行、
   Try=容错、Wait=人工等待）。
4. 设契约：每步输出必须结构化（schema），供下一步消费。
5. 设护栏：每个循环设步数与 Token 预算上限；每个外部调用设超时与重试策略。
6. 设可靠性：每个 Activity 幂等；长耗时 Activity 发心跳；失败路径写补偿（Saga）。
7. 设人工节点：审阅/批准作为一等节点，配 timer 与超时升级路径。
8. 校验：静态校验 + 一致性用例 + 重放验证。
9. 上线：审批后发布；记录版本号与变更日志。
10. 观测：埋点（耗时、成功率、重试次数、schedule-to-start latency），形成看板。

## 验证与证据要求
- 证据包：工作流定义与版本、静态校验报告、一致性用例与结果、
  重放验证记录、人工节点清单、补偿路径说明、变更日志。
- 每个外部调用必须说明：幂等方式、超时、重试策略、失败后的补偿动作。
- 每个人工节点必须说明：审阅人角色、超时时长、超期升级路径。
- 每个循环必须说明：步数上限、Token 上限、超限后的行为。

## 失败与升级策略
| 失败 | 处置 |
|---|---|
| 静态校验失败 | 回退定义，修正后重校验；不得绕过校验发布 |
| Activity 非幂等导致重复生效 | 立即停止，评估影响，补幂等键后重跑 |
| 长耗时 Activity 无心跳超时 | 改为带心跳的实现；重试从上一检查点续跑 |
| 人工节点超时 | 按既定路径升级提醒；不得自动放行 |
| 补偿失败 | 记录并升级；不得静默忽略 |
| 队列积压 | 按 schedule-to-start latency 扩缩；GPU 队列限并发 |
- 每个循环必须有步数上限与 Token 预算上限；超限即停并升级。

## 安全与合规红线
- 不得把密钥写入工作流定义、参数或日志（引擎对越权密钥抛 authorization 错误，HTTP 403）。
- 未加密的 payload 不得包含敏感数据（默认明文存储需配 Data Converter 加密）。
- 不得删除或篡改 Event History 与审计日志。
- 不得在生产环境启动未审批的工作流定义。
- 不得修改正在运行的工作流定义。

## 禁止事项
- 禁止把必须由 LLM 运行时决策的控制流硬编码为 workflow（反之亦然）。
- 禁止写出无超时的人工等待节点。
- 禁止省略补偿路径；禁止静默忽略补偿失败。
- 禁止混用 CPU / IO / GPU 队列。
- 禁止编造任务类型、错误码 URI、SDK 语言与版本号。
- 禁止使用 XX / XXX / ___ 等非标准占位符（统一用 [待填写] / [待核实]）。
- 禁止使用 emoji 与署名。

## 输出格式
- 工作流定义：带语言标记的 DSL 代码块 + 版本号 + 变更日志。
- 节点清单：节点 / 任务类型 / 输入 schema / 输出 schema / 超时 / 重试 / 幂等键 / 补偿。
- 人工节点清单：节点 / 审阅角色 / 超时 / 升级路径。
- 数值带单位；范围用 ~ 连接；中文全角标点；中英文之间加空格。

## 评估与自检
- 九项自检：控制流归属正确 / 每步有 schema / 每循环有预算上限 /
  每个 Activity 幂等 / 长耗时有心跳 / 人工节点有超时 / 补偿路径完整 /
  编号与错误码可核实 / 信息缺口已声明。
- 每次定义变更后跑一致性用例与重放验证；失败视为缺陷。

4.2. SKILL.md Specification

The following is the complete, copy-ready industry-standard SKILL.md text for the Workflow direction.

---
name: workflow-author
description: 设计、生成与校验工作流定义——选择编排模式、映射任务类型、设定结构化输出契约与预算护栏、设计人工介入节点与 Saga 补偿路径，并输出一致性用例与可观测性埋点方案。当用户要求"编排这个流程""把这个多步任务写成工作流""加一个人工审批节点""设计重试与补偿""工作流为什么卡住"时触发。
version: 1.0
created: 2026-09-12
---

# 工作流编排定义（Workflow Authoring）

## 适用场景
- 把多步骤任务编排为可持久执行的工作流。
- 为既有工作流增加人工介入、重试、补偿或并行分支。
- 工作流卡死或重试风暴的归因与修复。
- 生成一致性测试用例与可观测性埋点方案。

## 前置条件
- 已加载本方向 AGENTS.md；目标引擎与 DSL 版本已确定。
- 已明确 SLA / 超时要求与依赖服务契约（接口、幂等性、错误码）。
- 已明确人工介入节点的审批人与超时策略。
- 生产发布审批路径已确定。

## 输入
- 任务描述与步骤清单、目标引擎与 DSL 版本
- 依赖服务契约（schema、幂等、错误码）
- SLA / 超时 / 并发要求
- 可选：既有工作流定义、故障复盘记录

## 输出
- 工作流定义（DSL 代码块 + 版本号）
- 节点清单：任务类型 / 输入 schema / 输出 schema / 超时 / 重试 / 幂等键 / 补偿
- 人工节点清单：审阅角色 / 超时 / 升级路径
- 护栏说明：循环步数上限、Token 上限、并发限制
- 一致性用例与埋点方案

## 执行步骤
1. 判定控制流归属：可预定义 → workflow；需运行时决策 → agent 节点。
2. 选模式：优先 Prompt Chaining；需要分类用 Routing；需要提速或多视角用
   Parallelization（sectioning / voting）；子任务不可预知用 Orchestrator-workers；
   有明确评估标准且迭代有效用 Evaluator-optimizer。
3. 映射任务类型：Switch（路由）、For（批量）、Fork（并行）、Try（容错）、
   Wait（人工等待）、Listen（事件）、Run（脚本/服务）、Emit（事件输出）等。
4. 设契约：每步输入输出必须有 schema；无 schema 的步骤不予发布。
5. 设护栏：循环设步数与 Token 上限；外部调用设超时与指数退避重试。
6. 设可靠性：Activity 幂等（唯一约束 / upsert / 幂等令牌）；长耗时发心跳并带进度。
7. 设人工节点：一等节点 + timer + 超时升级路径，禁止无超时等待。
8. 设补偿：为每个不可逆外部调用写 Saga 补偿动作。
9. 校验：静态校验 + 一致性用例 + 重放验证；通过后提交审批。
10. 埋点：耗时、成功率、重试次数、schedule-to-start latency。

## 质量标准（DoD）
- 控制流归属判定正确，未把 agent 问题硬编码为 workflow。
- 每个步骤有结构化 schema；每个循环有预算上限。
- 每个 Activity 幂等；长耗时 Activity 有心跳。
- 每个不可逆调用有补偿动作；补偿失败有升级路径。
- 每个人工节点有审阅角色与超时升级路径。
- 一致性用例与重放验证通过；证据包齐备。

## 常见失败与处理
| 失败 | 根因 | 处置 |
|---|---|---|
| 重试风暴 | 无退避或重试无上限 | 改指数退避 + 最大次数 + 熔断 |
| 重复生效 | Activity 非幂等 | 补幂等键 / 唯一约束 / upsert |
| 长任务误判失败 | 无心跳，等待执行超时 | 改心跳上报，重试从检查点续跑 |
| 卡在人工节点 | 无 timer 与升级 | 补 timer 与到期升级路径 |
| 补偿缺失 | 不可逆调用未设计补偿 | 补 Saga 补偿；无法补偿的步骤前置审批 |
| 队列互相拖累 | CPU/IO/GPU 混用同一队列 | 按负载类型分队列，GPU 独占限并发 |

## 示例
用户请求：把"合同审批"编排为工作流，需支持人工审批与超时升级。
执行：
1. 控制流可预定义 → workflow；"判断是否需法务介入"由模型决定 → 用 Switch + agent 节点。
2. 模式：Prompt Chaining + Routing（按合同金额路由到不同审批人）。
3. 任务映射：Set（装配上下文）→ Switch（金额分级）→ Wait（人工审批）→ Try（归档+通知）。
4. schema：每步输出 JSON，含 contract_id、amount、approver、decision、timestamp。
5. 护栏：审批循环最多 3 轮；单轮 Token 上限 8K。
6. 可靠性：归档步骤用 contract_id 作幂等键；通知步骤发心跳。
7. 人工节点：审批人角色 + 48 小时 timer + 到期升级至其上级。
8. 补偿：归档失败则删除已写记录并回滚状态；通知失败仅记录不回滚。
9. 校验：静态校验 + 3 条一致性用例（正常通过 / 驳回 / 超时升级）+ 重放验证。
约束：Wait 节点必须配 timer；不得无超时等待；生产发布需审批。

4.3. Landing Checklist

5. Summary

The core judgment of the Workflow direction is: the reliability of AI applications comes from the orchestration layer, not the model layer.

The five most important conclusions:

1. Distinguish workflow from agent. Anthropic's official criterion: control flow predefined by code is a workflow, control flow decided autonomously by the LLM is an agent (R1). Confusing the two leads to "uncertain where it should be certain, and rigid where it should be flexible".
2. Durable execution is a hard requirement for AI workflows. Long-running AI tasks with many external dependencies, without state persistence and replay, will re-run the whole chain on a single hiccup. Temporal's core promise — "capture state at every step... pick up exactly where they left off" — is precisely the answer to this need.
3. AI workflows need four extra constraints: structured output, loop budget caps, human-in-the-loop as a first-class node, and persistent state with a replayable trace. These four are not present in traditional workflows.
4. Reliability patterns are enumerable: Activity idempotency, heartbeats for long tasks, queues separated by load, payload encryption with mTLS, and Saga compensation. These are not optimization items; they are prerequisites for going live.
5. Workflow-ization amplifies call volume by an order of magnitude. A ServiceNow executive stated that assist call volume in agentic-workflow scenarios is 5~12 times what it was (R2), so cost guardrails must be involved from the design stage.

Information gaps to note: there is currently no formal international standard for AI Agent workflows. CNCF Serverless Workflow is a general workflow standard rather than one specific to AI, but because its 12 task types happen to cover routing, bulk fan-out, parallelism, fault tolerance, and human waiting, it can serve as a reusable engineering baseline.

Information Gap Statement

6. References

1. Serverless Workflow Specification — CNCF / serverlessworkflow. https://github.com/serverlessworkflow/specification
2. Building Effective Agents — Anthropic Research. https://www.anthropic.com/research/building-effective-agents
3. Temporal — Durable Execution official site. https://temporal.io/
4. Trendyol Database Scaling with Temporal — Temporal official case study. https://temporal.io/resources/case-studies/trendyol-database-scaling-temporal
5. Scaling Renewals Engine with Temporal Workflows — Coalition engineering blog. https://www.coalitioninc.com/fr-ca/blog/broker-education/scaling-renewals-engine-temporal-workflows
6. ServiceNow FY2025 Q3 Earnings Call — The Globe and Mail. https://www.theglobeandmail.com/investing/markets/stocks/NOW/pressreleases/35791955/
7. Production-grade orchestration reliability patterns — Xgrid. https://www.xgrid.co/resources?p=11487
8. Apache KIE SonataFlow — Serverless Workflow runtime implementation. https://sonataflow.org/
9. Lemline — Serverless Workflow runtime implementation. https://github.com/lemline/lemline
10. Model Context Protocol Specification Changelog — MCP. https://modelcontextprotocol.io/specification/2025-06-18/changelog
11. AGENTS.md — A simple, open format for guiding coding agents. https://agents.md/
12. Celonis Process Intelligence / Forrester TEI (process-side reference) — Celonis. https://www.celonis.com/news/press/celonis-customers-saw-payback-in-6-months-and-383-roi