vaquarkhan/data-engineering-agent-skills
Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
ARCHETYPE
Orchestrator
A bench of specialised subagents. The main agent mostly delegates.
Copy this rig
# review before running: this installs third-party code
$ npx degit vaquarkhan/data-engineering-agent-skills/.claude ./rig-data-engineering-agent-skills # inspect, then merge into .claude/ MCP servers are added to Claude Code at local scope; env vars are shown as YOUR_… placeholders — we never store values. Files are fetched with degit into a separate folder so you can review before merging.
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/airflow-and-workflow-orchestration .claude/skills/airflow-and-workflow-orchestration $ npx degit vaquarkhan/data-engineering-agent-skills/skills/apache-beam-unified-batch-and-stream .claude/skills/apache-beam-unified-batch-and-stream $ npx degit vaquarkhan/data-engineering-agent-skills/skills/apache-hudi-lakehouse .claude/skills/apache-hudi-lakehouse $ npx degit vaquarkhan/data-engineering-agent-skills/skills/api-and-saas-ingestion-patterns .claude/skills/api-and-saas-ingestion-patterns $ npx degit vaquarkhan/data-engineering-agent-skills/skills/avro-protobuf-json-schema-registry .claude/skills/avro-protobuf-json-schema-registry $ npx degit vaquarkhan/data-engineering-agent-skills/skills/bigquery-and-dataform-platform-engineering .claude/skills/bigquery-and-dataform-platform-engineering $ npx degit vaquarkhan/data-engineering-agent-skills/skills/cdc-and-incremental-loading .claude/skills/cdc-and-incremental-loading $ npx degit vaquarkhan/data-engineering-agent-skills/skills/clickhouse-real-time-analytics .claude/skills/clickhouse-real-time-analytics $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-catalog-and-discovery .claude/skills/data-catalog-and-discovery $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-contract-testing-with-schema-registry .claude/skills/data-contract-testing-with-schema-registry $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-lake-and-zone-architecture .claude/skills/data-lake-and-zone-architecture $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-mesh-and-domain-oriented-design .claude/skills/data-mesh-and-domain-oriented-design $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-migration-and-platform-cutover .claude/skills/data-migration-and-platform-cutover $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-observability-and-sla-management .claude/skills/data-observability-and-sla-management $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-platform-ci-cd-and-release-management .claude/skills/data-platform-ci-cd-and-release-management $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-platform-disaster-recovery-and-business-continuity .claude/skills/data-platform-disaster-recovery-and-business-continuity $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-platform-operating-model-and-service-ownership .claude/skills/data-platform-operating-model-and-service-ownership $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-quality-and-contract-testing .claude/skills/data-quality-and-contract-testing $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-quality-platforms-and-rule-management .claude/skills/data-quality-platforms-and-rule-management $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-reconciliation-and-financial-controls .claude/skills/data-reconciliation-and-financial-controls $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-resiliency-testing-and-failure-injection .claude/skills/data-resiliency-testing-and-failure-injection $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-security-compliance-and-regulated-data .claude/skills/data-security-compliance-and-regulated-data $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-sharing-and-publishing-contracts .claude/skills/data-sharing-and-publishing-contracts $ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-specification .claude/skills/data-specification $ npx degit vaquarkhan/data-engineering-agent-skills/skills/dataplex-and-bigquery-governance .claude/skills/dataplex-and-bigquery-governance $ npx degit vaquarkhan/data-engineering-agent-skills/skills/dbt-and-analytics-engineering .claude/skills/dbt-and-analytics-engineering $ npx degit vaquarkhan/data-engineering-agent-skills/skills/debezium-and-kafka-connect-cdc .claude/skills/debezium-and-kafka-connect-cdc $ npx degit vaquarkhan/data-engineering-agent-skills/skills/delta-lake-and-medallion-architecture .claude/skills/delta-lake-and-medallion-architecture $ npx degit vaquarkhan/data-engineering-agent-skills/skills/duckdb-local-analytics-and-dev .claude/skills/duckdb-local-analytics-and-dev $ npx degit vaquarkhan/data-engineering-agent-skills/skills/enterprise-etl-and-data-integration-modernization .claude/skills/enterprise-etl-and-data-integration-modernization $ npx degit vaquarkhan/data-engineering-agent-skills/skills/esg-and-sustainability-regulatory-reporting .claude/skills/esg-and-sustainability-regulatory-reporting $ npx degit vaquarkhan/data-engineering-agent-skills/skills/etl-elt-and-modernization-strategy .claude/skills/etl-elt-and-modernization-strategy $ npx degit vaquarkhan/data-engineering-agent-skills/skills/feature-store-and-ml-data-pipelines .claude/skills/feature-store-and-ml-data-pipelines $ npx degit vaquarkhan/data-engineering-agent-skills/skills/file-and-partner-feed-ingestion .claude/skills/file-and-partner-feed-ingestion $ npx degit vaquarkhan/data-engineering-agent-skills/skills/glue-data-catalog-and-lake-formation-governance .claude/skills/glue-data-catalog-and-lake-formation-governance $ npx degit vaquarkhan/data-engineering-agent-skills/skills/great-expectations-deequ-and-cuallee .claude/skills/great-expectations-deequ-and-cuallee $ npx degit vaquarkhan/data-engineering-agent-skills/skills/incident-triage-and-pipeline-recovery .claude/skills/incident-triage-and-pipeline-recovery $ npx degit vaquarkhan/data-engineering-agent-skills/skills/java-data-engineering-and-integration-services .claude/skills/java-data-engineering-and-integration-services $ npx degit vaquarkhan/data-engineering-agent-skills/skills/kafka-resilience-and-schema-evolution .claude/skills/kafka-resilience-and-schema-evolution $ npx degit vaquarkhan/data-engineering-agent-skills/skills/lakefs-and-data-versioning .claude/skills/lakefs-and-data-versioning
$ curl -fsSL --create-dirs -o .claude/agents/analytics-engineer-reviewer.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/analytics-engineer-reviewer.md $ curl -fsSL --create-dirs -o .claude/agents/data-architect.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-architect.md $ curl -fsSL --create-dirs -o .claude/agents/data-platform-infrastructure-reviewer.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-platform-infrastructure-reviewer.md $ curl -fsSL --create-dirs -o .claude/agents/data-platform-reliability-reviewer.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-platform-reliability-reviewer.md $ curl -fsSL --create-dirs -o .claude/agents/data-security-and-compliance-auditor.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-security-and-compliance-auditor.md
This rig commits no guardrails. Here is the community baseline instead — the deny/ask rules most often found across all 16.2k rigs:
{
"permissions": {
"deny": [
"Read(./.env)",
"Read(./.env.*)",
"Read(**/.env)",
"Read(~/.ssh/**)",
"Bash(git push --force:*)",
"Read(**/*.pem)",
"Read(.env)",
"Read(~/.aws/**)",
"Read(**/*.key)",
"Read(**/.env.*)",
"Bash(git push --force*)",
"Bash(rm -rf:*)",
"Bash(rm -rf *)",
"Bash(sudo *)",
"Bash(sudo:*)",
"Bash(rm -rf /)",
"Bash(git push -f:*)",
"Read(.env.*)",
"Bash(git reset --hard*)",
"Bash(rm -rf /*)"
],
"ask": [
"Bash(git push:*)",
"Bash(git push *)",
"Bash(gh pr merge *)",
"Bash(git commit:*)",
"Bash(rm *)",
"Bash(gh pr merge:*)",
"Bash(gh pr create *)",
"Bash(git reset --hard:*)",
"Bash(npm publish:*)",
"Bash(git rebase *)"
]
}
} Skills (74)
airflow-and-workflow-orchestrationapache-beam-unified-batch-and-streamapache-hudi-lakehouseapi-and-saas-ingestion-patternsavro-protobuf-json-schema-registrybigquery-and-dataform-platform-engineeringcdc-and-incremental-loadingclickhouse-real-time-analyticsdata-catalog-and-discoverydata-contract-testing-with-schema-registrydata-lake-and-zone-architecturedata-mesh-and-domain-oriented-designdata-migration-and-platform-cutoverdata-observability-and-sla-managementdata-platform-ci-cd-and-release-managementdata-platform-disaster-recovery-and-business-continuitydata-platform-operating-model-and-service-ownershipdata-quality-and-contract-testingdata-quality-platforms-and-rule-managementdata-reconciliation-and-financial-controlsdata-resiliency-testing-and-failure-injectiondata-security-compliance-and-regulated-datadata-sharing-and-publishing-contractsdata-specificationdataplex-and-bigquery-governancedbt-and-analytics-engineeringdebezium-and-kafka-connect-cdcdelta-lake-and-medallion-architectureduckdb-local-analytics-and-deventerprise-etl-and-data-integration-modernizationesg-and-sustainability-regulatory-reportingetl-elt-and-modernization-strategyfeature-store-and-ml-data-pipelinesfile-and-partner-feed-ingestionglue-data-catalog-and-lake-formation-governancegreat-expectations-deequ-and-cualleeincident-triage-and-pipeline-recoveryjava-data-engineering-and-integration-serviceskafka-resilience-and-schema-evolutionlakefs-and-data-versioninglakehouse-table-format-engineeringlineage-pii-and-governancelower-environment-data-masking-and-obfuscationmainframe-modernization-and-data-offloadmaster-data-and-entity-resolutionmcp-data-observability-integrationmicrosoft-purview-and-azure-data-governancenotebook-to-production-hardeningopenmetadata-datahub-and-openlineageoperational-datastore-selection-relational-and-nosqlorchestration-and-backfillspipeline-planning-and-task-breakdownprivacy-retention-and-right-to-deletepython-data-engineering-and-pipeline-packagingregional-data-compliance-and-sovereigntyreverse-etl-and-operational-data-servingsafe-backfill-and-replay-orchestrationscala-data-engineering-on-jvm-runtimesschema-evolution-and-contract-migrationssemantic-layer-and-metric-governancesnowflake-native-pipelines-and-governancesource-reliability-and-extraction-resiliencespark-and-distributed-processingspark-serverless-reliability-and-state-managementstreaming-and-messaging-systemssuperset-and-metrics-servingterraform-and-data-platform-infrastructuretest-data-preparation-and-synthetic-datatrino-presto-federated-queryunity-catalog-and-lakehouse-governanceusing-data-agent-skillsusing-data-engineering-agent-skillswarehouse-and-schema-designwarehouse-performance-and-cost-optimization
Subagents (5)
| analytics-engineer-reviewer | — |
| data-architect | — |
| data-platform-infrastructure-reviewer | — |
| data-platform-reliability-reviewer | — |
| data-security-and-compliance-auditor | — |
Slash commands (8)
/backfill/build/plan/review/ship/spec/test/validate
Cursor rules (4)
00-data-agent-core · always10-spec-first20-platform-preset-selector30-quality-gates
Kiro (3 steering)
How Kiro files are scored: methodology.
Similar rigs
Dimkox/Getzilla
AI speed. Engineering discipline. Getzilla lets an AI write code for your project and checks that code before it is allowed to stay.
Orchestrator 13.7k tok · · Q25
mohamedaeldin/claude-code-setup
Claude Code best-practice playbook: slash commands and skills for mobile (iOS/Android) and web development
Orchestrator 3.6k tok · · Q3
bakhod1r/awesome-agents
109 specialist Claude Code subagents · 16 engineering teams · 11 commands · 10 skills. Architecture → build → verify → ship, with the delegation chain enforced. Installable as a plugin marketplace.
Orchestrator 4.2k tok · · Q20
rtk-ai/rtk
CLI proxy that reduces LLM token consumption by 60-90% on common dev commands. Single Rust binary, zero dependencies
Orchestrator 2.6k tok · · Q0