check_conformance

Instructions

Implementation Reference

• bawbel_mcp/server.py:288-337 (handler)

  The check_conformance function is the MCP tool handler that scores an MCP server manifest against the MCP specification. It accepts a URL or local file path, delegates to the bawbel CLI via _run_bawbel(['scan-conformance', url_or_path]), parses the result (score, grade, is_conformant, checks), categorizes checks into failed/passed/skipped, and returns a formatted human-readable conformance report.

  @mcp.tool()
  def check_conformance(url_or_path: str) -> str:
      """
      Score an MCP server manifest against the MCP specification.
  
      Runs 18 checks across 3 tiers (REQUIRED, RECOMMENDED, BEST PRACTICE)
      and returns a grade from A+ to F. A server is conformant when all
      REQUIRED checks pass. Grade F means at least one REQUIRED check failed.
  
      Accepts:
          - A local file path to a server.json manifest
          - A server base URL (fetches .well-known/mcp.json automatically)
  
      Args:
          url_or_path: Local path to server.json OR base URL of MCP server
      """
      result = _run_bawbel(["scan-conformance", url_or_path])
  
      if result.get("error"):
          return f"Error: {result['error']}"
  
      score = result.get("score", 0)
      grade = result.get("grade", "?")
      is_conformant = result.get("is_conformant", False)
      checks = result.get("checks", [])
  
      lines = [
          f"Conformance score: {score:.0f}/100  Grade: {grade}",
          f"Conformant: {'Yes' if is_conformant else 'No (REQUIRED check failed)'}",
          "",
      ]
  
      failed = [c for c in checks if c.get("status") == "FAIL"]
      passed = [c for c in checks if c.get("status") == "PASS"]
      skipped = [c for c in checks if c.get("status") == "SKIP"]
  
      if failed:
          lines.append(f"FAILED ({len(failed)}):")
          for c in failed:
              tier = c.get("tier", "")
              name = c.get("check_id", "")
              msg = c.get("message", "")
              lines.append(f"  [{tier}] {name}: {msg}")
          lines.append("")
  
      lines.append(
          f"Passed: {len(passed)}  Failed: {len(failed)}  Skipped: {len(skipped)}"
      )
  
      return "\n".join(lines)

• bawbel_mcp/server.py:54-102 (helper)

  The _run_bawbel helper function is used by check_conformance to execute the 'bawbel scan-conformance' CLI command. It runs the bawbel CLI with arguments, parses JSON output, and handles errors (timeout, parse errors, missing CLI).

  def _run_bawbel(args: list[str], input_file: Optional[str] = None) -> dict:
      """
      Run bawbel CLI and return parsed JSON output.
      Returns error dict on failure.
      """
      cmd = ["bawbel"] + args + ["--format", "json"]
      if input_file:
          cmd.append(input_file)
  
      try:
          result = subprocess.run(  # nosec B603  # noqa: S603
              cmd,
              capture_output=True,
              text=True,
              timeout=60,
          )
          raw = result.stdout.strip()
          if not raw:
              return {
                  "error": result.stderr.strip() or "Scanner produced no output",
                  "findings": [],
                  "toxic_flows": [],
                  "risk_score": 0,
              }
  
          # JSON output is a list of file results
          start = raw.find("[")
          if start < 0:
              return {"error": raw[:300], "findings": [], "toxic_flows": [], "risk_score": 0}
  
          results = json.loads(raw[start:])
          if results:
              return results[0]
          return {"findings": [], "toxic_flows": [], "risk_score": 0}
  
      except subprocess.TimeoutExpired:
          return {"error": "Scan timeout (60s)", "findings": [], "toxic_flows": [], "risk_score": 0}
      except json.JSONDecodeError as e:
          return {"error": f"Parse error: {e}", "findings": [], "toxic_flows": [], "risk_score": 0}
      except FileNotFoundError:
          return {
              "error": (
                  "bawbel CLI not found. "
                  "Install with: pip install bawbel-scanner"
              ),
              "findings": [],
              "toxic_flows": [],
              "risk_score": 0,
          }

• bawbel_mcp/server.py:36-45 (registration)

  The FastMCP server instance registers all tools via decorators. The instructions mention check_conformance as a tool to 'verify a server follows the MCP spec.' The actual tool registration happens via the @mcp.tool() decorator on line 288.

  mcp = FastMCP(
      name="Bawbel Scanner",
      instructions=(
          "Security scanner for MCP servers and agentic AI components. "
          "Use scan_server_card before connecting to any MCP server. "
          "Use scan_content to check skill files or system prompts. "
          "Use check_conformance to verify a server follows the MCP spec. "
          "Use lookup_ave or search_ave to query the AVE threat intelligence database."
      ),
  )

• bawbel_mcp/server.py:289-303 (schema)

  The function signature and docstring define the input schema: it accepts a single string parameter 'url_or_path' (local path to server.json or base URL of MCP server). The output is a formatted string with score, grade, conformant status, and list of failed/passed/skipped checks.

  def check_conformance(url_or_path: str) -> str:
      """
      Score an MCP server manifest against the MCP specification.
  
      Runs 18 checks across 3 tiers (REQUIRED, RECOMMENDED, BEST PRACTICE)
      and returns a grade from A+ to F. A server is conformant when all
      REQUIRED checks pass. Grade F means at least one REQUIRED check failed.
  
      Accepts:
          - A local file path to a server.json manifest
          - A server base URL (fetches .well-known/mcp.json automatically)
  
      Args:
          url_or_path: Local path to server.json OR base URL of MCP server
      """