Airflow 状态存储

原名：airflow-state-store

使用 Airflow 状态存储和可恢复任务机制保存跨重试、跨运行的进度。

中文 Skills 技能说明

适合外部作业提交、轮询、断点恢复和水位管理；先明确幂等键、状态生命周期和故障恢复，再替换容易丢失状态的实现。Airflow、OpenLineage、dbt Core 和本地数据工具可在用户自己的环境使用；Astronomer 托管能力须使用自有合法账户，并确认中国大陆网络、区域、费用和数据位置。执行前读取真实项目、版本和官方资料，输出目标、依据、影响、验证结果与回退方式；没有证据时明确说明未知，不编造数据、配置或成功状态。 创建、修改、部署、发布、迁移、发送、删除或产生费用的操作，必须先展示目标和差异并获得有权限人员确认。 代码、数据库、日志、凭据、邮件、客户和企业资料按敏感内容处理，只读取任务所需范围，上传或外发前完成授权、脱敏和最小化。

上游能力依据

上游原始适用说明：Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key/value stores (taskstatestore, assetstatestore) and the crash-safe ResumableJobMixin. Use when the user asks about task state store, checkpointing in tasks, persisting state across retries, job IDs surviving worker crashes, watermarks, asset metadata, resumable tasks, crash-safe operators, or "what's new in Airflow 3.3". Also use proactively when reading a DAG that uses Variables or XCom for intra-task coordination state — flag the anti-pattern and recommend taskstatestore or assetstatestore instead. Also use proactively when reviewing ANY DAG that submits a job to an external system and waits for it to finish — Databricks, Snowflake, BigQuery, Redshift, Spark, dbt Cloud, EMR, AWS Batch, etc. — whether that is one submit-and-wait operator or split across a separate submit task plus a sensor/polling task; this covers waitfortermination, deferrable, durable, hand-rolled sensors polling a run/job id, and whether to collapse a submit+sensor split into one task. The taskstatestore/assetstatestore/ResumableJobMixin state-persistence pieces require Airflow 3.3+; the submit+poll architecture guidance itself applies on any Airflow version — do not skip this skill for a pre-3.3 DAG.

上游 SKILL.md 主要章节（保留原文标题）：

• Section 1 — Pick the right primitive
• Section 2 — Detect anti-patterns in existing DAGs (on demand)
• Section 3 — taskstatestore: per-task coordination state
• Section 4 — assetstatestore: per-asset metadata across DAG runs
• Section 5 — ResumableJobMixin: crash-safe external job submission
• Implementing the mixin
• What happens on retry
• externalidkey warning
• Before (anti-pattern)
• Section 6 — Submit-and-poll DAGs: one task or two?

使用边界

先读取项目约定、依赖版本和现有测试，再提出最小改动。写文件、运行脚本、提交代码或发布前应检查差异并保留回退路径。

作者、翻译与许可证

原作者

Astronomer 与 astronomer/agents contributors

中文翻译

CEOFans翻译

许可证

Apache-2.0

上游来源

https://github.com/astronomer/agents/tree/dbd9bb79c4f7b7ac9c1cdd2b4b75213c81f423e5/skills/airflow-state-store

适用范围

平台：linux、macos、windows；标签：架构与规划、Airflow 状态存储

安全提示

基础静态扫描不等于绝对安全。技能可能调用命令、浏览器、云服务或本地文件，请在最小权限环境中使用，高风险操作必须人工确认。

如发现侵权、许可证或安全问题，可在本页前台提交投诉，管理员复核后可立即下架。