~ / rigs / brainfish-ai / webcrawl-anything

brainfish-ai/webcrawl-anything

Give your AI agent the whole web. Free, local, no API key. Search, scrape, map, crawl and monitor any site, PDF, YouTube video or CVE from Claude Code, Cursor, Codex or the CLI.

↗ GitHub ★ 1 MIT updated 14d ago project Claude Code
field: AI / ML workflow: DocsResearch
share on X
ARCHETYPE
Pragmatist
A balanced, no-drama setup: some rules, some tools, nothing extreme.
CONTEXT TAX · EVERY TURN
up to ~3.2k tokens
Claude Code loads MCP tools on demand, so the real cost is usually lower.
Moderate · median rig: 2.3k · breakdown
GUARDRAILS
0/5
No committed guardrails · details
Quality 0/100 Guardrails v2

Copy this rig

# review before running: this installs third-party code
$ claude mcp add --transport http webcrawl https://webcrawl.app/mcp

MCP servers are added to Claude Code at local scope; env vars are shown as YOUR_… placeholders — we never store values. Files are fetched with degit into a separate folder so you can review before merging.

$ claude mcp add --transport http webcrawl https://webcrawl.app/mcp
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-crawl .claude/skills/webcrawl-crawl
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-deal-watch .claude/skills/webcrawl-deal-watch
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-jobs-leads .claude/skills/webcrawl-jobs-leads
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-learn-topic .claude/skills/webcrawl-learn-topic
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-protected-sites .claude/skills/webcrawl-protected-sites
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-real-estate .claude/skills/webcrawl-real-estate
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-research-corpus .claude/skills/webcrawl-research-corpus
$ npx degit brainfish-ai/webcrawl-anything/.agents/skills/webcrawl-ticket-watch .claude/skills/webcrawl-ticket-watch
$ npx degit brainfish-ai/webcrawl-anything/plugins/chatgpt/skills/get-started .claude/skills/get-started

This rig commits no guardrails. Here is the community baseline instead — the deny/ask rules most often found across all 16.2k rigs:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(**/.env)",
      "Read(~/.ssh/**)",
      "Bash(git push --force:*)",
      "Read(**/*.pem)",
      "Read(.env)",
      "Read(~/.aws/**)",
      "Read(**/*.key)",
      "Read(**/.env.*)",
      "Bash(git push --force*)",
      "Bash(rm -rf:*)",
      "Bash(rm -rf *)",
      "Bash(sudo *)",
      "Bash(sudo:*)",
      "Bash(rm -rf /)",
      "Bash(git push -f:*)",
      "Read(.env.*)",
      "Bash(git reset --hard*)",
      "Bash(rm -rf /*)"
    ],
    "ask": [
      "Bash(git push:*)",
      "Bash(git push *)",
      "Bash(gh pr merge *)",
      "Bash(git commit:*)",
      "Bash(rm *)",
      "Bash(gh pr merge:*)",
      "Bash(gh pr create *)",
      "Bash(git reset --hard:*)",
      "Bash(npm publish:*)",
      "Bash(git rebase *)"
    ]
  }
}

MCP servers (1)

serversourceest. tokens
webcrawl.app · "webcrawl" remote · remote 2.5k

Skills (9)

Similar rigs

copied ✓