~ / rigs / vaquarkhan / data-engineering-agent-skills

vaquarkhan/data-engineering-agent-skills

Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

share on X
ARCHETYPE
Orchestrator
A bench of specialised subagents. The main agent mostly delegates.
CONTEXT TAX · EVERY TURN
~7.8k tokens
Heavy · median rig: 2.3k · breakdown
GUARDRAILS
0/5
No committed guardrails · details
Quality 0/100 Guardrails v2

Copy this rig

# review before running: this installs third-party code
$ npx degit vaquarkhan/data-engineering-agent-skills/.claude ./rig-data-engineering-agent-skills  # inspect, then merge into .claude/

MCP servers are added to Claude Code at local scope; env vars are shown as YOUR_… placeholders — we never store values. Files are fetched with degit into a separate folder so you can review before merging.

$ npx degit vaquarkhan/data-engineering-agent-skills/skills/airflow-and-workflow-orchestration .claude/skills/airflow-and-workflow-orchestration
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/apache-beam-unified-batch-and-stream .claude/skills/apache-beam-unified-batch-and-stream
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/apache-hudi-lakehouse .claude/skills/apache-hudi-lakehouse
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/api-and-saas-ingestion-patterns .claude/skills/api-and-saas-ingestion-patterns
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/avro-protobuf-json-schema-registry .claude/skills/avro-protobuf-json-schema-registry
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/bigquery-and-dataform-platform-engineering .claude/skills/bigquery-and-dataform-platform-engineering
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/cdc-and-incremental-loading .claude/skills/cdc-and-incremental-loading
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/clickhouse-real-time-analytics .claude/skills/clickhouse-real-time-analytics
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-catalog-and-discovery .claude/skills/data-catalog-and-discovery
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-contract-testing-with-schema-registry .claude/skills/data-contract-testing-with-schema-registry
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-lake-and-zone-architecture .claude/skills/data-lake-and-zone-architecture
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-mesh-and-domain-oriented-design .claude/skills/data-mesh-and-domain-oriented-design
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-migration-and-platform-cutover .claude/skills/data-migration-and-platform-cutover
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-observability-and-sla-management .claude/skills/data-observability-and-sla-management
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-platform-ci-cd-and-release-management .claude/skills/data-platform-ci-cd-and-release-management
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-platform-disaster-recovery-and-business-continuity .claude/skills/data-platform-disaster-recovery-and-business-continuity
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-platform-operating-model-and-service-ownership .claude/skills/data-platform-operating-model-and-service-ownership
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-quality-and-contract-testing .claude/skills/data-quality-and-contract-testing
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-quality-platforms-and-rule-management .claude/skills/data-quality-platforms-and-rule-management
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-reconciliation-and-financial-controls .claude/skills/data-reconciliation-and-financial-controls
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-resiliency-testing-and-failure-injection .claude/skills/data-resiliency-testing-and-failure-injection
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-security-compliance-and-regulated-data .claude/skills/data-security-compliance-and-regulated-data
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-sharing-and-publishing-contracts .claude/skills/data-sharing-and-publishing-contracts
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/data-specification .claude/skills/data-specification
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/dataplex-and-bigquery-governance .claude/skills/dataplex-and-bigquery-governance
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/dbt-and-analytics-engineering .claude/skills/dbt-and-analytics-engineering
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/debezium-and-kafka-connect-cdc .claude/skills/debezium-and-kafka-connect-cdc
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/delta-lake-and-medallion-architecture .claude/skills/delta-lake-and-medallion-architecture
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/duckdb-local-analytics-and-dev .claude/skills/duckdb-local-analytics-and-dev
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/enterprise-etl-and-data-integration-modernization .claude/skills/enterprise-etl-and-data-integration-modernization
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/esg-and-sustainability-regulatory-reporting .claude/skills/esg-and-sustainability-regulatory-reporting
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/etl-elt-and-modernization-strategy .claude/skills/etl-elt-and-modernization-strategy
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/feature-store-and-ml-data-pipelines .claude/skills/feature-store-and-ml-data-pipelines
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/file-and-partner-feed-ingestion .claude/skills/file-and-partner-feed-ingestion
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/glue-data-catalog-and-lake-formation-governance .claude/skills/glue-data-catalog-and-lake-formation-governance
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/great-expectations-deequ-and-cuallee .claude/skills/great-expectations-deequ-and-cuallee
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/incident-triage-and-pipeline-recovery .claude/skills/incident-triage-and-pipeline-recovery
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/java-data-engineering-and-integration-services .claude/skills/java-data-engineering-and-integration-services
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/kafka-resilience-and-schema-evolution .claude/skills/kafka-resilience-and-schema-evolution
$ npx degit vaquarkhan/data-engineering-agent-skills/skills/lakefs-and-data-versioning .claude/skills/lakefs-and-data-versioning
$ curl -fsSL --create-dirs -o .claude/agents/analytics-engineer-reviewer.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/analytics-engineer-reviewer.md
$ curl -fsSL --create-dirs -o .claude/agents/data-architect.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-architect.md
$ curl -fsSL --create-dirs -o .claude/agents/data-platform-infrastructure-reviewer.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-platform-infrastructure-reviewer.md
$ curl -fsSL --create-dirs -o .claude/agents/data-platform-reliability-reviewer.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-platform-reliability-reviewer.md
$ curl -fsSL --create-dirs -o .claude/agents/data-security-and-compliance-auditor.md https://raw.githubusercontent.com/vaquarkhan/data-engineering-agent-skills/main/agents/data-security-and-compliance-auditor.md

This rig commits no guardrails. Here is the community baseline instead — the deny/ask rules most often found across all 16.2k rigs:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(**/.env)",
      "Read(~/.ssh/**)",
      "Bash(git push --force:*)",
      "Read(**/*.pem)",
      "Read(.env)",
      "Read(~/.aws/**)",
      "Read(**/*.key)",
      "Read(**/.env.*)",
      "Bash(git push --force*)",
      "Bash(rm -rf:*)",
      "Bash(rm -rf *)",
      "Bash(sudo *)",
      "Bash(sudo:*)",
      "Bash(rm -rf /)",
      "Bash(git push -f:*)",
      "Read(.env.*)",
      "Bash(git reset --hard*)",
      "Bash(rm -rf /*)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(git push *)",
      "Bash(gh pr merge *)",
      "Bash(git commit:*)",
      "Bash(rm *)",
      "Bash(gh pr merge:*)",
      "Bash(gh pr create *)",
      "Bash(git reset --hard:*)",
      "Bash(npm publish:*)",
      "Bash(git rebase *)"
    ]
  }
}

Skills (74)

airflow-and-workflow-orchestrationapache-beam-unified-batch-and-streamapache-hudi-lakehouseapi-and-saas-ingestion-patternsavro-protobuf-json-schema-registrybigquery-and-dataform-platform-engineeringcdc-and-incremental-loadingclickhouse-real-time-analyticsdata-catalog-and-discoverydata-contract-testing-with-schema-registrydata-lake-and-zone-architecturedata-mesh-and-domain-oriented-designdata-migration-and-platform-cutoverdata-observability-and-sla-managementdata-platform-ci-cd-and-release-managementdata-platform-disaster-recovery-and-business-continuitydata-platform-operating-model-and-service-ownershipdata-quality-and-contract-testingdata-quality-platforms-and-rule-managementdata-reconciliation-and-financial-controlsdata-resiliency-testing-and-failure-injectiondata-security-compliance-and-regulated-datadata-sharing-and-publishing-contractsdata-specificationdataplex-and-bigquery-governancedbt-and-analytics-engineeringdebezium-and-kafka-connect-cdcdelta-lake-and-medallion-architectureduckdb-local-analytics-and-deventerprise-etl-and-data-integration-modernizationesg-and-sustainability-regulatory-reportingetl-elt-and-modernization-strategyfeature-store-and-ml-data-pipelinesfile-and-partner-feed-ingestionglue-data-catalog-and-lake-formation-governancegreat-expectations-deequ-and-cualleeincident-triage-and-pipeline-recoveryjava-data-engineering-and-integration-serviceskafka-resilience-and-schema-evolutionlakefs-and-data-versioninglakehouse-table-format-engineeringlineage-pii-and-governancelower-environment-data-masking-and-obfuscationmainframe-modernization-and-data-offloadmaster-data-and-entity-resolutionmcp-data-observability-integrationmicrosoft-purview-and-azure-data-governancenotebook-to-production-hardeningopenmetadata-datahub-and-openlineageoperational-datastore-selection-relational-and-nosqlorchestration-and-backfillspipeline-planning-and-task-breakdownprivacy-retention-and-right-to-deletepython-data-engineering-and-pipeline-packagingregional-data-compliance-and-sovereigntyreverse-etl-and-operational-data-servingsafe-backfill-and-replay-orchestrationscala-data-engineering-on-jvm-runtimesschema-evolution-and-contract-migrationssemantic-layer-and-metric-governancesnowflake-native-pipelines-and-governancesource-reliability-and-extraction-resiliencespark-and-distributed-processingspark-serverless-reliability-and-state-managementstreaming-and-messaging-systemssuperset-and-metrics-servingterraform-and-data-platform-infrastructuretest-data-preparation-and-synthetic-datatrino-presto-federated-queryunity-catalog-and-lakehouse-governanceusing-data-agent-skillsusing-data-engineering-agent-skillswarehouse-and-schema-designwarehouse-performance-and-cost-optimization

Subagents (5)

analytics-engineer-reviewer—
data-architect—
data-platform-infrastructure-reviewer—
data-platform-reliability-reviewer—
data-security-and-compliance-auditor—

Slash commands (8)

/backfill/build/plan/review/ship/spec/test/validate

Cursor rules (4)

00-data-agent-core · always10-spec-first20-platform-preset-selector30-quality-gates

Kiro (3 steering)

How Kiro files are scored: methodology.

Similar rigs

copied ✓