# <img src="assets/lilfetch-icon.svg" width="64" height="64"> lilFetch
An MCP (Model Context Protocol) server that scrapes webpages using [crawl4ai](https://github.com/unclecode/crawl4ai) and Playwright for more robust scraping. Tested on CoPilot in VSCode but may work for others.
## Features
- Enables html and/or text scraping of one or more urls directly in your chat prompt. Use the scraped response in followup queries for market research summarization, context for new file creation, etc.
- Leverages Playwright and a headless instance of Chromium to load JS heavy sites and web apps where basic `#fetch` and `curl` commands fall short.
- Strong focus on minimal commands and configuration to install and get scraping.
## Prerequisites
Before installing, ensure:
- **Node.js 14+**: Download from [nodejs.org](https://nodejs.org) or via Homebrew (`brew install node` on macOS).
- **Python 3.10+**: Auto-detected during setup. Install from [python.org](https://python.org) or Homebrew (`brew install python@3.12` on macOS). If using pyenv, set a 3.10+ version active (`pyenv global 3.12.0`).
- **First Run Time**: Setup downloads ~200MB (Playwright browsers) and takes 1-2 minutes.
## Installation
Install globally for use across workspaces, or restrict to local installation if you just want to test in this repo or enhance it further.
### Option 1: Global Install
For running `npx lilfetch` from any directory (portable CLI).
1. Clone the Repo
```
git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp
```
2. Install Globally
```
npm run global-install
```
- Sets up Python venv in `~/.lilfetch-venv` (user-wide).
3. Configure in Any VS Code Workspace (add to `.vscode/mcp.json` or global MCP settings):
```jsonc
{
"servers": {
"lilFetch": {
"type": "stdio",
"command": "npx",
"args": ["lilfetch"]
}
}
}
```
4. Test It
- In new terminal window, run `npx lilfetch` to start MCP server.
- In VS Code, prompt with something like...
```
Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.
```
### Option 2: Local Install
For testing/extending in the repo.
1. Clone the Repo
```
git clone https://github.com/jphdevsf/lilfetch-mcp.git lilfetch-mcp
cd lilfetch-mcp
```
2. Install Locally
```
npm install
```
- Sets up `./node_modules/lilfetch/` and `.bin/lilfetch`.
- Python venv in repo `.venv` (local to this project).
3. MCP.json Workspace Configuration
Navigate to `.vscode/mcp.json` (create if missing) and add:
```jsonc
{
"servers": {
"lilFetch": {
"type": "stdio",
"command": "node",
"args": ["bin/lilfetch.js"]
}
}
}
```
**Note:** Ensure `bin/lilfetch.js` is executable: Run `chmod +x bin/lilfetch.js` in the terminal.
4. Test It
- In new terminal window, navigate to this repo and run `npm run dev` or `./node_modules/.bin/lilfetch`.
- In VS Code, prompt with something like...
```
Use lilFetch to scrape top news headlines from www.cnn.com and write to a markdown file in root of my repo.
```
## Uninstallation
To fully remove a global installation (including the npm package and Python virtual environment):
### Global Uninstall
From the repo directory (or anywhere):
```
npm run global-uninstall
```
- This runs `npm uninstall -g lilfetch` to remove the global npm package and binary.
- Followed by `rm -rf ~/.lilfetch-venv` to delete the user-wide Python venv (including installed deps and Playwright browsers).
- **Warning**: The `rm -rf` command is irreversible. It only affects the `.lilfetch-venv` directory in your home folder. Back up if needed (unlikely).
### Manual Uninstall (Alternative)
1. Remove npm package:
```
npm uninstall -g lilfetch
```
2. Remove Python venv:
```
rm -rf ~/.lilfetch-venv
```
- On Windows: `rmdir /s /q %USERPROFILE%\.lilfetch-venv`
### Local Uninstall
For local installs (e.g., after `npm install`):
```
npm uninstall
rm -rf .venv
```
- This removes the local Node modules and repo-specific Python venv.
### Verification
- `npm list -g --depth=0` (no `lilfetch`).
- `which lilfetch` (empty).
- `ls ~/.lilfetch-venv` (no such file).
For local: `rm -rf node_modules .venv` and verify no `./.bin/lilfetch`.
## Development
- Edit `mcp_server.py` for Python logic.
- Update `bin/lilfetch.js` for wrapper changes.
- Bump version in `package.json`, then `npm run pack`.
- For global testing: `npm install -g .` then `npx lilfetch`.
## Troubleshooting
- **Permission Errors (Global Install)**: See Prerequisites for user-owned NPM setup. Avoid sudo—use the config steps.
- **Python Not Found/Version Error**: Ensure Python 3.8+ is in PATH. For pyenv: `pyenv install 3.12.0 && pyenv global 3.12.0`, then re-run install. Check: `python3 --version`.
- **Venv/Deps Fail**: For local: Delete `.venv` and re-run `npm install`. For global: Delete `~/.lilfetch-venv` and re-run `npm install -g .`. Manual fix (local): `python3 -m venv .venv && .venv/bin/pip install -r requirements.txt && .venv/bin/python -m playwright install`. Manual fix (global): `python3 -m venv ~/.lilfetch-venv && ~/.lilfetch-venv/bin/pip install -r requirements.txt && ~/.lilfetch-venv/bin/python -m playwright install`.
- **Playwright Browsers Missing**: Run `python -m playwright install` in the venv (or manually as logged).
- **MCP Not Detected in VS Code**: Restart VS Code after config; ensure workspace is open correctly.
- **Uninstall**:
- Global: `npm uninstall -g lilfetch` + `rm -rf ~/.lilfetch-venv`.
- Local: `rm -rf node_modules package-lock.json .venv`.
૮₍ • ᴥ • ₎ა
License: MIT
