1. Habit
  2. Skills
  3. Tavily Crawl skill
tavily-ai · 17K installs

Tavily Crawl skill
Runs by itself, every Monday early.

Fresh local copy of docs.

Install the Tavily Crawl skill and make it a habit: follow https://gethabit.dev/skills/tavily-ai/tavily-crawl.md

  1. 1Copy
  2. 2Paste into Claude Code, Codex or Cursor
  3. 3It runs by itself

17K installs488 starsMIT licenceSource on GitHub ↗

Tavily Crawl

by tavily-ai · 17K installs

The skill

The tavily crawl skill crawls a website through the Tavily CLI and extracts content from many pages, with depth, breadth and path filters, and can save each page as a local markdown file.

Only when you ask.

The habit

Fresh local copy of docs.

↻ Runs by itselfevery Monday early

How the habit works.
You don't have to.

Three questionsWhen does it start? How does it check its work? What does it remember?

01

It starts by itself

every Monday early. Runs in Claude Code routine.

  • ↻ every Monday early
  • 0 5 * * 1

02

It checks its work

crawl output check. If a check fails, it tells you.

  • ✓tvly crawl finishes without an authentication error
  • ✓the output directory holds more than zero .md files
  • ✓the page count stays at or under --limit

03

It remembers

crawl log. The next run reads it first: last crawl URL, path filters and page count.

loops/docs-crawl-sync-loop/runs.md

One run, step by step.
Then it stops.

Stop ruleStops after one site and at most 50 pages per run.

  1. 01

    Check the CLI

    If tvly is missing or not signed in, the run stops and leaves login to the user.

  2. 02

    Map first

    It uses tavily-map to see the site structure before a full crawl.

  3. 03

    Set limits

    It starts with --max-depth 1 and a --limit, and scales up only when needed.

  4. 04

    Crawl

    It runs tvly crawl with --select-paths for the docs section.

  5. 05

    Save pages

    It uses --output-dir to write each page as a markdown file.

  6. 06

    Log

    It records the page count and compares it with the last run.

What it may do.
Nothing more.

Safe by defaultIt writes only in its own folder and asks you before anything goes out.

Is it for you?
Use it when it fits.

Quick checkRead both lists. If the left one sounds like you, the Tavily Crawl skill helps.

Use it when

  • ✓ Downloading a docs section
  • ✓ Bulk-extracting pages from one domain
  • ✓ Saving a site as markdown

Skip it when

  • – You need a single page
  • – You do not have a URL yet

Questions

Tavily Crawl skill, answered.

Which site does it crawl?

Replace docs.example.com, the example in the skill, with the docs you want to keep.

I want context for an agent, not files.

Use --instructions with --chunks-per-source to get only the relevant chunks instead of full pages.

How does it sign in unattended?

With a securely provided TAVILY_API_KEY, or the user signs in once with tvly login.

How do I install the Tavily Crawl skill?

Copy the prompt at the top into your agent. It installs the skill from tavily-ai/skills and sets up the habit. Or run npx skills add tavily-ai/skills --skill tavily-crawl.

Is the Tavily Crawl skill free?

Yes. It is published under MIT. Habit is free too.

Tavily Crawl. Now as a habit.

Install the Tavily Crawl skill and make it a habit: follow https://gethabit.dev/skills/tavily-ai/tavily-crawl.md

  • Free
  • No account
  • Undo in one line

All tavily-ai skills →More docs skills →

Works in

  • Claude Code
  • Codex
  • Cursor
  • ChatGPT
  • Claude app

Made by tavily-ai, published under MIT. Habit links to the original and never changes it. Is this your skill?

Copying didn't work here. Select the line and copy it by hand.