~ / rigs / aws-samples / sample-Agentic-Ai-Data-Operations

aws-samples/sample-Agentic-Ai-Data-Operations

Agentic Data Onboarding Platform(ADOP) — an AI-driven system that automates Silver→Bronze→Gold data pipeline lifecycle (ETL scripts, quality checks, Airflow DAGs, semantic layer) from a natural language description, reducing weeks of

↗ GitHub ★ 36 MIT-0 updated 1mo ago project Claude Code Claude plugin
stack: pytestPython
share on X
ARCHETYPE
MCP Hoarder
Six or more MCP servers wired in. Every tool, always in context.
CONTEXT TAX · EVERY TURN
up to ~36.5k tokens
Claude Code loads MCP tools on demand, so the real cost is usually lower.
Context hog · median rig: 2.3k · breakdown
GUARDRAILS
2/5
Pre-tool screening hook · No YOLO mode · details
Quality 29/100 Guardrails v2

Copy this rig

# review before running: this installs third-party code
$ claude mcp add glue-athena -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with fastmcp --with boto3 --python 3.13 mcp-servers/glue-athena-server/server.py
$ claude mcp add lakeformation -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with fastmcp --with boto3 --python 3.13 mcp-servers/lakeformation-server/server.py
$ claude mcp add sagemaker-catalog -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with fastmcp --with boto3 --python 3.13 mcp-servers/sagemaker-catalog-server/server.py
$ claude mcp add core -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-core-mcp-server awslabs.core-mcp-server
$ claude mcp add iam -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-iam-mcp-server awslabs.iam-mcp-server
$ claude mcp add lambda -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-lambda-mcp-server awslabs.lambda-mcp-server
$ claude mcp add s3-tables -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-s3-tables-mcp-server awslabs.s3-tables-mcp-server
$ claude mcp add cloudtrail -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-cloudtrail-mcp-server awslabs.cloudtrail-mcp-server
$ claude mcp add redshift -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-redshift-mcp-server awslabs.redshift-mcp-server
$ claude mcp add cloudwatch -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-cloudwatch-mcp-server awslabs.cloudwatch-mcp-server
$ claude mcp add cost-explorer -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-cost-explorer-mcp-server awslabs.cost-explorer-mcp-server
$ claude mcp add dynamodb -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-dynamodb-mcp-server awslabs.dynamodb-mcp-server
$ claude mcp add pii-detection -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with boto3 --with fastmcp --python 3.13 mcp-servers/pii-detection-server/server.py
$ npx degit aws-samples/sample-Agentic-Ai-Data-Operations/.claude ./rig-sample-agentic-ai-data-operations  # inspect, then merge into .claude/

MCP servers are added to Claude Code at local scope; env vars are shown as YOUR_… placeholders — we never store values. Files are fetched with degit into a separate folder so you can review before merging.

$ claude mcp add glue-athena -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with fastmcp --with boto3 --python 3.13 mcp-servers/glue-athena-server/server.py
$ claude mcp add lakeformation -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with fastmcp --with boto3 --python 3.13 mcp-servers/lakeformation-server/server.py
$ claude mcp add sagemaker-catalog -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with fastmcp --with boto3 --python 3.13 mcp-servers/sagemaker-catalog-server/server.py
$ claude mcp add core -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-core-mcp-server awslabs.core-mcp-server
$ claude mcp add iam -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-iam-mcp-server awslabs.iam-mcp-server
$ claude mcp add lambda -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-lambda-mcp-server awslabs.lambda-mcp-server
$ claude mcp add s3-tables -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-s3-tables-mcp-server awslabs.s3-tables-mcp-server
$ claude mcp add cloudtrail -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-cloudtrail-mcp-server awslabs.cloudtrail-mcp-server
$ claude mcp add redshift -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-redshift-mcp-server awslabs.redshift-mcp-server
$ claude mcp add cloudwatch -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-cloudwatch-mcp-server awslabs.cloudwatch-mcp-server
$ claude mcp add cost-explorer -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-cost-explorer-mcp-server awslabs.cost-explorer-mcp-server
$ claude mcp add dynamodb -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -e FASTMCP_LOG_LEVEL=YOUR_FASTMCP_LOG_LEVEL -- uvx --from awslabs-dynamodb-mcp-server awslabs.dynamodb-mcp-server
$ claude mcp add pii-detection -e AWS_PROFILE=YOUR_AWS_PROFILE -e AWS_REGION=YOUR_AWS_REGION -- uv run --no-project --with boto3 --with fastmcp --python 3.13 mcp-servers/pii-detection-server/server.py
$ npx degit aws-samples/sample-Agentic-Ai-Data-Operations/plugins/claude-code/skills/data-compliance .claude/skills/data-compliance
$ npx degit aws-samples/sample-Agentic-Ai-Data-Operations/plugins/claude-code/skills/decision-engine .claude/skills/decision-engine
$ npx degit aws-samples/sample-Agentic-Ai-Data-Operations/plugins/claude-code/skills/discovery-protocol .claude/skills/discovery-protocol
$ npx degit aws-samples/sample-Agentic-Ai-Data-Operations/plugins/claude-code/skills/observability .claude/skills/observability
$ npx degit aws-samples/sample-Agentic-Ai-Data-Operations/plugins/claude-code/skills/pipeline-conventions .claude/skills/pipeline-conventions
$ curl -fsSL --create-dirs -o .claude/agents/dag-agent.md https://raw.githubusercontent.com/aws-samples/sample-Agentic-Ai-Data-Operations/main/.claude/agents/dag-agent.md
$ curl -fsSL --create-dirs -o .claude/agents/iac-agent.md https://raw.githubusercontent.com/aws-samples/sample-Agentic-Ai-Data-Operations/main/.claude/agents/iac-agent.md
$ curl -fsSL --create-dirs -o .claude/agents/metadata-agent.md https://raw.githubusercontent.com/aws-samples/sample-Agentic-Ai-Data-Operations/main/.claude/agents/metadata-agent.md
$ curl -fsSL --create-dirs -o .claude/agents/ontology-agent.md https://raw.githubusercontent.com/aws-samples/sample-Agentic-Ai-Data-Operations/main/.claude/agents/ontology-agent.md
$ curl -fsSL --create-dirs -o .claude/agents/quality-agent.md https://raw.githubusercontent.com/aws-samples/sample-Agentic-Ai-Data-Operations/main/.claude/agents/quality-agent.md
$ curl -fsSL --create-dirs -o .claude/agents/transformation-agent.md https://raw.githubusercontent.com/aws-samples/sample-Agentic-Ai-Data-Operations/main/.claude/agents/transformation-agent.md

Merge into .claude/settings.json (project) or ~/.claude/settings.json (user). Hook commands reference scripts in the source repo — copy those too.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": ".claude/hooks/check-discovery-gate.sh"
          },
          {
            "type": "command",
            "command": ".claude/hooks/check-logging.sh"
          }
        ]
      },
      {
        "matcher": "Write|Edit|MultiEdit",
        "hooks": [
          {
            "type": "command",
            "command": "python3 .claude/hooks/enforce_template_codegen.py"
          }
        ]
      }
    ]
  }
}

MCP servers (13)

serversourceest. tokens
fastmcp · "glue-athena"
env: AWS_PROFILE, AWS_REGION
pypi 2.5k
fastmcp · "lakeformation"
env: AWS_PROFILE, AWS_REGION
pypi 2.5k
fastmcp · "sagemaker-catalog"
env: AWS_PROFILE, AWS_REGION
pypi 2.5k
awslabs-core-mcp-server · "core"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-iam-mcp-server · "iam"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-lambda-mcp-server · "lambda"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-s3-tables-mcp-server · "s3-tables"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-cloudtrail-mcp-server · "cloudtrail"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-redshift-mcp-server · "redshift"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-cloudwatch-mcp-server · "cloudwatch"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-cost-explorer-mcp-server · "cost-explorer"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
awslabs-dynamodb-mcp-server · "dynamodb"
env: AWS_PROFILE, AWS_REGION, FASTMCP_LOG_LEVEL
pypi 2.5k
boto3 · "pii-detection"
env: AWS_PROFILE, AWS_REGION
pypi 2.5k

Skills (5)

Subagents (6)

dag-agent
model: sonnet
Builds the DAG spec and renders the workload's Airflow DAG via the deterministic codegen renderer. Spawned by the Data Onboarding Agent at Step 4.5. Produces dags/{workload}_dag.py plus unit and integ
iac-agent
model: opus
Converts a built workload's artifacts (Glue PySpark, MWAA DAG, YAML configs, quality rules, Cedar policies) into deployable Terraform / CDK Python / CDK TypeScript / CloudFormation plus an APPLY_GUIDE
metadata-agent
model: opus
Extracts, infers, classifies and catalogs metadata for a workload's data source. Spawned by the Data Onboarding Agent in Phase 3 (profiling) and Step 4.2 (formalize catalog). Produces config/source.ya
ontology-agent
model: opus
Induces an OWL2 ontology and R2RML mappings from a workload's semantic.yaml plus its Glue Gold-zone schema, validates the Turtle, and stages three artifacts locally for AWS Semantic Layer handoff. Spa
quality-agent
model: opus
Builds the quality spec (5 dimensions, rules, gates, anomaly detection) and renders quality check scripts via the deterministic codegen renderer. Spawned by the Data Onboarding Agent at Step 4.4. Prod
transformation-agent
model: opus
Produces Bronze/Silver/Gold transform specs and renders AWS Glue ETL (PySpark + Iceberg) scripts via the deterministic codegen renderer. Spawned by the Data Onboarding Agent at Step 4.3. Produces conf

Hooks (5)

eventmatcherruns
PreToolUseEdit|Write.claude/hooks/check-discovery-gate.sh
PreToolUseEdit|Write.claude/hooks/check-logging.sh
PreToolUseWrite|Edit|MultiEditpython3 .claude/hooks/enforce_template_codegen.py
SessionStart*python3 .claude/hooks/inject_run_context.py
PostToolUseAskUserQuestionpython3 .claude/hooks/log_conversation.py

Slash commands (3)

/devops-workflow/onboard-workflow/security-review

Similar rigs

copied ✓