Skip to main content
Glama
Sign In
doc-scraper

doc-scraper

by Sriram-PR
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Go
Local

get_freshness

Read-onlyIdempotent

Check a site's last crawl summary, age, output/state presence, and running job to decide whether to query existing crawl data or run crawl_site first.

Instructions

Return the most recent crawl summary for a site (last_crawl with its start and end times, total_pages and mode, plus age_seconds), output/state dir presence and any running job. Use this to decide whether to query the existing crawl or run crawl_site first.

Input Schema

NameRequiredDescriptionDefault
site_keyYesSite key from config (use list_sites to discover available keys)

Output Schema

NameRequiredDescriptionDefault
site_keyYes
last_crawlNoLatest recorded crawl; absent if none
output_dirYes
age_secondsNoSeconds since last_crawl ended; present with last_crawl
running_jobNo
next_actionsYes
output_existsYes
state_dir_existsYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed6 schema fields changedv2.24.1
    • addedOutput schema / properties / running_job / properties / failures
      Added value: +{
+  "description": "Failed pages by class (temporary, gone, selector, policy, other), each with a count and one sample",
+  "items": {
+    "properties": {
+      "class": {
+        "type": "string"
+      },
+      "count": {
+        "type": "integer"
+      },
+      "sample_error": {
+        "type": "string"
+      },
+      "sample_url": {
+        "type": "string"
+      }
+    },
+    "required": [
+      "class",
+      "count",
+      "sample_url",
+      "sample_error"
+    ],
+    "type": "object"
+  },
+  "type": [
+    "null",
+    "array"
+  ]
+}
    • addedOutput schema / properties / running_job / properties / pages_carried_forward
      Added value: +{
+  "description": "Pages that failed temporarily and kept their copy from the previous crawl",
+  "type": "integer"
+}
    • addedOutput schema / properties / running_job / properties / pages_failed
      Added value: +{
+  "description": "Pages that failed; noindex pages are neither saved nor failed",
+  "type": "integer"
+}
    • addedOutput schema / properties / running_job / properties / pages_saved
      Added value: +{
+  "description": "Pages written by this crawl",
+  "type": "integer"
+}
    • addedOutput schema / properties / running_job / properties / refused
      Added value: +{
+  "description": "Set when a finished crawl was kept back instead of published: empty, temporary_failures or shrink",
+  "type": "string"
+}
    • changedOutput schema / properties / running_job / required
      Previous value: -[
-  "job_id",
-  "site_key",
-  "status",
-  "started_at",
-  "pages_processed",
-  "pages_queued",
-  "pages_unchanged",
-  "incremental"
-]New value: +[
+  "job_id",
+  "site_key",
+  "status",
+  "started_at",
+  "pages_processed",
+  "pages_queued",
+  "pages_unchanged",
+  "pages_saved",
+  "pages_failed",
+  "pages_carried_forward",
+  "incremental"
+]
  2. Changed1 schema field changedv2.21.0
    • addedOutput schema / properties / running_job / properties / pages_queued / description
      Added value: +"Pages waiting to be crawled; for a cancelled job, the pages left uncrawled"
  3. Changed1 schema field changedv2.15.0
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
+  "properties": {
+    "age_seconds": {
+      "description": "Seconds since last_crawl ended; present with last_crawl",
+      "type": [
+        "null",
+        "integer"
+      ]
+    },
+    "last_crawl": {
+      "description": "Latest recorded crawl; absent if none",
+      "properties": {
+        "crawl_ended_at": {
+          "type": "string"
+        },
+        "crawl_id": {
+          "type": "integer"
+        },
+        "crawl_started_at": {
+          "type": "string"
+        },
+        "mode": {
+          "description": "full or incremental",
+          "type": "string"
+        },
+        "total_pages": {
+          "type": "integer"
+        }
+      },
+      "required": [
+        "crawl_id",
+        "crawl_started_at",
+        "crawl_ended_at",
+        "total_pages",
+        "mode"
+      ],
+      "type": [
+        "null",
+        "object"
+      ]
+    },
+    "next_actions": {
+      "type": "string"
+    },
+    "output_dir": {
+      "type": "string"
+    },
+    "output_exists": {
+      "type": "boolean"
+    },
+    "running_job": {
+      "properties": {
+        "completed_at": {
+          "type": "string"
+        },
+        "duration_seconds": {
+          "description": "Present once completed_at is",
+          "type": [
+            "null",
+            "number"
+          ]
+        },
+        "error_message": {
+          "type": "string"
+        },
+        "incremental": {
+          "type": "boolean"
+        },
+        "job_id": {
+          "type": "string"
+        },
+        "pages_processed": {
+          "type": "integer"
+        },
+        "pages_queued": {
+          "type": "integer"
+        },
+        "pages_unchanged": {
+          "description": "Incremental re-checks whose content had not changed; not counted in pages_processed",
+          "type": "integer"
+        },
+        "site_key": {
+          "type": "string"
+        },
+        "started_at": {
+          "type": "string"
+        },
+        "status": {
+          "description": "pending, running, completed, failed or cancelled",
+          "type": "string"
+        }
+      },
+      "required": [
+        "job_id",
+        "site_key",
+        "status",
+        "started_at",
+        "pages_processed",
+        "pages_queued",
+        "pages_unchanged",
+        "incremental"
+      ],
+      "type": [
+        "null",
+        "object"
+      ]
+    },
+    "site_key": {
+      "type": "string"
+    },
+    "state_dir_exists": {
+      "type": "boolean"
+    }
+  },
+  "required": [
+    "site_key",
+    "output_dir",
+    "output_exists",
+    "state_dir_exists",
+    "next_actions"
+  ],
+  "type": "object"
+}
  4. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive, closed-world behavior, so the description's added value is the decision-relevant content: age_seconds, dir presence, and any running job. It goes beyond restating annotations by describing what state the agent will see, though it doesn't say how freshness should be interpreted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the return payload and closing with the decision use case. Dense but nothing is wasted; the parenthetical field list is long yet earned.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Complete for a single-parameter read tool with annotations and an output schema: the agent knows what it returns and why to call it. The only gap is guidance on interpreting age_seconds to judge staleness, which the agent must infer.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single site_key parameter is fully documented in the schema, including a pointer to list_sites. The description adds no parameter-level detail, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('return the most recent crawl summary for a site') and enumerates the payload (last_crawl start/end, total_pages, mode, age_seconds, dir presence, running job). This is clearly distinguishable from crawl_site, get_job_status, and other siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly frames usage as a decision step: 'Use this to decide whether to query the existing crawl or run crawl_site first,' naming the alternative tool. It never states when not to use it or what threshold makes a crawl stale, but the routing intent is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools