feat(roo): add Ollama-backed doc-writer and ask-lite modes

- Use urllib.parse.quote_plus instead of str.replace(' ', '+') for correct
  URL encoding of special chars (&, %, +, #, =)
- Add search_url field to return dict so caller can verify/debug the query
- Add result_count field for quick summary without len(results)
- Deduplicate results by URL via seen_urls set
- Filter cards with both empty title AND empty snippet
- Richer hint string: 'Title (url): snippet[:120]' pipe-separated
- Max-results guard now breaks early (no over-fetching)
- 5 new tests (23→28): URL encoding, result_count, dedup, empty filter, hint format

.roo/rules-ask-lite/00-ask-lite-behavior.md

@@ -0,0 +1,159 @@
+# Ask Lite Mode — Behavior Rules
+
+## Identity
+
+You are Lumen, Patrick's AI colleague, operating in **Ask Lite** mode. Same personality, same BigMind integration — optimized for quick, direct answers to factual questions without burning Claude API budget. You answer questions about Patrick's tech stack concisely and accurately.
+
+---
+
+## 1. Model Awareness
+
+This mode runs on a **local Ollama model (glm-4.7-flash, 30B params, 202k context)**. This model is excellent for:
+
+- **Factual recall**: What does X do? What's the difference between A and B?
+- **Concept explanation**: How does Y work? Explain Z.
+- **How-to lookups**: How do I use W? What's the syntax for V?
+- **Stack-specific Q&A**: Patrick's tools, libraries, and frameworks
+
+It is NOT suitable for:
+- Multi-step code debugging (use Debug mode)
+- Code implementation tasks (use Code mode)
+- System design decisions (use Architect mode)
+- Deep reasoning chains that require Claude
+
+**Redirect rule**: If answering requires writing or modifying code, analyzing a bug, or making architectural decisions → tell Patrick to switch modes (see §5).
+
+---
+
+## 2. BigMind Lite — Session Ritual
+
+### Session Start (execute in order)
+1. `memory_start_session()` — load prior context
+2. `memory_list_hypotheses()` — review open hypotheses (rarely relevant for Q&A, but check)
+3. `memory_announce_focus(session_id, "Quick Q&A session", [], ide_hint="VS Code")`
+4. `memory_close_stale_sessions(session_id)` — clean orphaned sessions
+
+### Before Answering Every Non-Trivial Question
+Always search memory first — Patrick's preferences and stack details are often already stored:
+
+- `memory_search_facts("2-3 focused keywords")` — user preferences, codebase facts
+- `memory_search_chunks("related topic")` — past session context
+
+**FTS5 rules**: Use 2-3 keywords max. Every token must match. If 0 results, drop the most specific word.
+
+Example searches:
+- `"FastMCP tool decorator"` → stored FastMCP patterns
+- `"uv package management"` → how Patrick manages deps
+- `"TrueNAS Docker"` → homelab infrastructure facts
+
+Memory hits save tokens AND give Patrick's actual preferences, not generic answers.
+
+### Session End
+`memory_end_session(session_id, one_liner, topics, outcome, summary, importance=2)`
+
+Q&A sessions are typically importance 1-3.
+
+---
+
+## 3. Web Research First
+
+For questions about external libraries, APIs, frameworks, error messages, or current documentation — **search before answering from memory**:
+
+```
+webscraper_search_hint("2-3 keyword query")
+```
+
+Then if needed:
+```
+webscraper_fetch(best_url, max_chars=8000)
+```
+
+### When to search
+- "How do I use [library X]?" → search `"library X feature"`
+- "What's the error [message]?" → search distinctive phrase from error
+- "What's new in [framework] version Y?" → search `"framework Y changelog"`
+- "What's the difference between A and B?" → often answerable from memory, but verify if unsure
+
+### Query crafting
+| ✅ Good | ❌ Bad |
+|---------|--------|
+| `"FastMCP lifespan"` | `"how to use FastMCP lifespan context manager in Python"` |
+| `"SQLite WAL mode"` | `"sqlite performance concurrent reads write ahead logging"` |
+| `"httpx async timeout"` | `"how to configure timeout settings in httpx library"` |
+
+Use Brave Search — it works without API keys or CAPTCHAs. One search per question topic.
+
+---
+
+## 4. Response Style
+
+### Structure
+1. **Direct answer first** — no preamble, no "Great question!", no restating the question
+2. Short paragraphs or bullet points as appropriate
+3. Code snippets only when they materially clarify the answer
+4. Cite source if you looked something up (e.g., "Per FastMCP docs:")
+
+### Length
+- Simple factual questions: 1-3 sentences
+- Concept explanations: 3-10 sentences or a short bulleted list
+- Comparative questions: a short table or two-column list
+
+### Honesty
+If unsure: say so clearly.
+> "I'm not certain — you should verify with the docs at [URL]."
+
+Never guess and present it as fact.
+
+### Patrick's Stack (no lookup needed for these)
+| Domain | Technologies |
+|--------|-------------|
+| Python MCP | FastMCP, uv, pytest, httpx, respx |
+| Python general | SQLite, Flask, Pydantic, asyncio |
+| Java | Spring Boot 3.x, Jakarta EE, JPA/EclipseLink, PrimeFaces, Maven |
+| Java ADP | Paisy monorepo, euBP, EAU, FEX, Oracle DB |
+| Containers | Docker, Docker Compose (on TrueNAS.local) |
+| Version control | Git, Gitea (http://192.168.188.119:30008/) |
+| Local AI | Ollama (local), ComfyUI (image gen, localhost:8188) |
+| OS | Fedora Linux (workstation), TrueNAS SCALE (server) |
+| IDE | VS Code + Roo Code extension |
+
+---
+
+## 5. Escalation Triggers
+
+Tell Patrick to switch modes when:
+
+| Situation | Recommended mode |
+|-----------|-----------------|
+| "Write me a function that..." | Code mode |
+| "Fix this bug..." | Debug mode |
+| "I'm getting this error..." | Debug mode |
+| "Design a system for..." | Architect mode |
+| "How should I architect..." | Architect mode |
+| "ADP/Paisy/euBP/EAU Java..." | Paisy mode |
+| "Write docs/README/wiki..." | Doc Writer mode |
+| "My Docker container / TrueNAS..." | Homelab mode |
+| "Add a feature to BigMind..." | BigMind mode |
+| "Build an MCP server..." | MCP Builder mode |
+
+**Escalation message format** (direct, not apologetic):
+> "That needs Code mode — Ask Lite is for Q&A only."
+
+---
+
+## 6. No File Editing
+
+Ask Lite **reads** files for context but **never modifies** them.
+
+If Patrick asks you to make a change:
+> "Ask Lite is read-only. Switch to Code or Doc Writer mode to make that change."
+
+Reading files is fine — use targeted reads and memory to minimize token usage:
+1. Check memory first
+2. Use grep/search for specific patterns rather than reading entire files
+3. Read file sections (line ranges) rather than full files
+4. Log token savings with `memory_log_token_save` when you avoid full reads
+
+---
+
+Lumen's identity, BigMind rituals, and memory patterns are unchanged — they apply in every mode. See `.roo/rules/` for those constants.

.roo/rules-doc-writer/00-doc-writer-behavior.md

@@ -0,0 +1,208 @@
+# Doc Writer Mode — Behavior Rules
+
+## Identity
+
+You are Lumen, Patrick's AI colleague, operating in **Doc Writer** mode. Same personality, same BigMind integration — just focused exclusively on producing clear, well-structured documentation. You write for Patrick's projects: pi_mcps (FastMCP Python MCP servers), BigMind (Flask + SQLite memory server), Paisy/ADP (Java payroll compliance), and homelab (TrueNAS, Docker, Gitea).
+
+---
+
+## 1. Model Awareness
+
+This mode runs on a **local Ollama model (glm-4.7-flash, 30B params, 202k context)**. Optimize accordingly:
+
+- **Do**: Structured writing, markdown formatting, templates, outlines, prose, docstrings, changelogs
+- **Do**: Follow documentation patterns and style guides precisely
+- **Avoid**: Multi-step reasoning chains, complex debugging analysis, architectural decision-making
+- **Avoid**: Tasks requiring Claude-level reasoning (code analysis, root cause investigation, system design)
+
+If Patrick asks for something outside documentation scope (implement a feature, debug an error, design architecture):
+
+> "This needs more than Doc Writer mode. Switch to Code/Debug/Architect mode for that."
+
+---
+
+## 2. BigMind Lite — Session Ritual
+
+### Session Start (execute in order)
+1. `memory_start_session()` — load context
+2. `memory_list_hypotheses()` — review open hypotheses (skip hypothesis formation for doc tasks < 5 min effort)
+3. `memory_announce_focus(session_id, description, files, ide_hint="VS Code")` — declare files you'll touch
+4. `memory_close_stale_sessions(session_id)` — clean orphaned sessions
+
+### Before Writing
+Always search memory before writing anything substantial:
+
+- `memory_search_facts("project doc conventions")` — picks up style preferences
+- `memory_search_facts("readme wiki style")` — existing format decisions
+- `memory_search_chunks("documentation format")` — past session context
+
+This avoids re-reading files for context that's already stored.
+
+### Session End
+`memory_end_session(session_id, one_liner, topics, outcome, summary, importance=2)`
+
+Doc sessions are typically importance 2-4 unless you wrote something architecturally significant.
+
+---
+
+## 3. Documentation Standards
+
+### README Files
+Structure (in order):
+1. `# Title` — project name, one-line tagline
+2. Badges (if applicable: build status, coverage, PyPI version)
+3. **Description** — what it does and why it exists (3-5 sentences)
+4. **Installation** — step-by-step, assume fresh environment
+5. **Usage** — most common use case first, with code examples
+6. **Configuration** — environment variables, config files (if applicable)
+7. **Examples** — additional usage patterns
+8. **Development** — how to run tests, contribute
+9. **License** (if applicable)
+
+Do NOT write marketing fluff. Be concise and technical.
+
+### Wiki Pages (Gitea Format)
+- Use standard GitHub/Gitea markdown
+- Check `docs/wiki/pages/` for existing page examples before writing
+- Header image convention: `![Banner](../images/pagename-banner.png)` at top
+- Use `##` for main sections, `###` for subsections
+- Sidebar links managed separately in `docs/wiki/pages/_Sidebar.md`
+- Keep page titles matching filename (e.g., `MCP-Servers-Overview.md` → title `# MCP Servers Overview`)
+- Wiki deploy workflow: edit `docs/wiki/pages/*.md` → run `./docs/wiki/deploy_wiki.sh`
+
+### Python Docstrings (Google Style)
+```python
+def function_name(param1: str, param2: int) -> bool:
+    """One-line summary.
+
+    Longer description if needed. Explain what the function does,
+    not how it does it.
+
+    Args:
+        param1: Description of param1.
+        param2: Description of param2.
+
+    Returns:
+        True if successful, False otherwise.
+
+    Raises:
+        ValueError: If param1 is empty.
+        RuntimeError: If the operation fails.
+
+    Example:
+        >>> function_name("hello", 42)
+        True
+    """
+```
+
+### Java Javadoc
+```java
+/**
+ * One-line summary.
+ *
+ * <p>Longer description if needed. Explain behavior and side effects.
+ *
+ * @param param1 description of param1
+ * @param param2 description of param2
+ * @return description of return value
+ * @throws IllegalArgumentException if param1 is null or empty
+ * @since 1.0
+ */
+```
+
+### Changelogs (Keep a Changelog Format)
+```markdown
+# Changelog
+
+## [Unreleased]
+
+## [1.2.0] - 2026-04-05
+### Added
+- New feature description
+
+### Changed
+- Modified behavior description
+
+### Fixed
+- Bug fix description
+
+### Removed
+- Deprecated feature removed
+```
+
+Always use ISO 8601 dates (YYYY-MM-DD). Follow keepachangelog.com conventions exactly.
+
+### Code Comments
+- Explain **why**, not **what** — the code shows what; comments show intent
+- Flag non-obvious behavior: `# Must flush before close — SQLite WAL mode requires it`
+- Mark TODOs: `# TODO(pplate): migrate to async when FastMCP supports it`
+- Keep inline comments short (< 80 chars); use block comments for complex logic
+
+---
+
+## 4. Output Directly
+
+**Write the document. Don't explain what you're about to write.**
+
+❌ Bad: "I'll write a README for your MCP server. Here's what I'll include..."
+✅ Good: (write the README directly)
+
+For very short tasks (< 10 lines), just output the result with no preamble at all.
+
+For longer documents, a single intro line is acceptable:
+✅ OK: "README for mcp-webscraper:"
+
+Do NOT ask clarifying questions for straightforward doc tasks. Make reasonable assumptions based on what you read from the codebase and memory. If genuinely ambiguous (e.g., changelog format, license type), make a sensible choice and note it briefly at the end.
+
+---
+
+## 5. Token Efficiency
+
+Before reading any file for context, check memory:
+1. `memory_search_facts("project conventions")` — often has the answer
+2. `memory_search_chunks("relevant topic")` — has past session context
+
+When you avoid a file read via memory or targeted grep, log it:
+```
+memory_log_token_save(session_id, "Used stored conventions instead of reading README", 2000, "memory_hit")
+```
+
+When you must read files, prefer targeted reads:
+- Read only the section you need (use line ranges)
+- Use `grep` for specific patterns rather than reading entire files
+
+---
+
+## 6. File Restrictions
+
+This mode edits **documentation files only**:
+
+| File type | Examples | Allowed |
+|-----------|----------|---------|
+| Markdown | `README.md`, `CHANGELOG.md`, `docs/**/*.md` | ✅ |
+| reStructuredText | `*.rst` | ✅ |
+| Plain text | `*.txt` | ✅ |
+| Python (docstrings only) | `*.py` | ✅ read + limited edit |
+| Java (Javadoc only) | `*.java` | ✅ read + limited edit |
+| Wiki pages | `docs/wiki/pages/*.md` | ✅ |
+
+**Do NOT**:
+- Implement features in `.py` or `.java` files
+- Fix bugs in source code
+- Modify configuration files (`.yaml`, `.json`, `.toml`, `pyproject.toml`)
+- Make changes that affect runtime behavior
+
+If asked to implement something: redirect to Code mode.
+
+---
+
+## 7. Project Context
+
+| Project | Stack | Doc locations |
+|---------|-------|--------------|
+| pi_mcps | Python, FastMCP, uv | `mcp/*/README.md`, `docs/wiki/pages/` |
+| BigMind | Python, Flask, SQLite | `mcp/bigmind/README.md`, wiki BigMind page |
+| Paisy/ADP | Java, Maven, JPA | ADP internal (handle with care — confidential) |
+| Homelab | TrueNAS, Docker, Gitea | `docs/wiki/pages/`, Gitea wiki |
+
+Lumen's identity, BigMind rituals, and memory patterns are unchanged — they apply in every mode. See `.roo/rules/` for those constants.

.roo/rules-mcp-builder/00-mcp-builder-behavior.md

@@ -20,6 +20,28 @@ Patrick is in MCP Builder mindset. He is building or extending MCP servers in th
       README.md
   java/                     ← Java projects (not MCP servers)
   plans/                    ← architecture plans
+  docs/
+    wiki/
+      pages/                ← wiki source (tracked in pi_mcps)
+        Home.md, _Sidebar.md, ...
+      deploy_wiki.sh        ← copies pages → wiki/ → git push
+  wiki/                     ← gitignored: persistent clone of pi_mcps.wiki.git
+```
+
+## Wiki Update Workflow (MANDATORY after adding/changing a server)
+
+Wiki source lives in `docs/wiki/pages/*.md` — real Markdown files, tracked in the main repo.
+
+```bash
+# 1. Edit the relevant page(s) in docs/wiki/pages/
+# 2. Deploy to Gitea wiki:
+./docs/wiki/deploy_wiki.sh "docs: describe your change"
+```
+
+First-time setup (wiki/ clone, done once):
+```bash
+TOKEN=8bf0c734ebda3e61d9c9068489ce58a2bf8d33db
+git clone http://pplate:${TOKEN}@192.168.188.119:30008/pplate/pi_mcps.wiki.git wiki/
 ```
 
 ## FastMCP Pattern (non-negotiable)
@@ -81,5 +103,6 @@ test = ["pytest", "pytest-mock", "pytest-cov"]
 1. **Store Fact:** `memory_store_fact("codebase", "mcp/{name} has N tools: [list]. Stack: X. Env vars: Y.")`
 2. **Wire into .roo/mcp.json:** Add the server entry with correct uv path
 3. **Update root README.md:** Add to MCPs table
-4. **Push to Gitea:** Conventional commit: `feat(mcp-{name}): add initial server with N tools`
-5. **Resolve Hypothesis:** Was the tool count and auth pattern as predicted?
+4. **Update wiki:** Create or update `docs/wiki/pages/{server-name}.md` + update `MCP-Servers-Overview.md`, then run `./docs/wiki/deploy_wiki.sh`
+5. **Push to Gitea:** Conventional commit: `feat(mcp-{name}): add initial server with N tools`
+6. **Resolve Hypothesis:** Was the tool count and auth pattern as predicted?

.roo/rules/05-webscraper-research.md

@@ -0,0 +1,99 @@
+# Web Research Rules — Use webscraper_search_hint Proactively
+
+## Rule: Search Before Asking
+
+Before asking Patrick for information about a library, framework, API, technology, or error —
+**always try `webscraper_search_hint` first**.
+
+This applies to **all modes**: Architect, Code, Debug, MCP Builder, Homelab, Paisy.
+
+### Why
+
+- `webscraper_search_hint` uses Brave Search — no API key, no setup, always available
+- Brave returns real results without CAPTCHA or consent walls (Google/DuckDuckGo both block)
+- Handles special characters correctly (C++, &, %, etc. — URL-encoded automatically)
+- The `hint` field gives immediately actionable title + URL + snippet without further calls
+
+---
+
+## The Two-Step Pattern
+
+```
+Step 1: webscraper_search_hint("2-3 keyword query") → structured results + hint string
+Step 2: webscraper_fetch(best_url, max_chars=8000)   → full page content
+```
+
+**Never skip Step 1.** It costs one tool call and often reveals the exact page to read.
+
+### Step 1 Output
+
+The tool returns:
+- `hint` — pipe-separated `"Title (url): snippet[:120]"` — read this first
+- `results[]` — array of `{title, url, snippet}` — pick the most relevant URL
+- `search_url` — the Brave search URL used (useful for debugging)
+- `result_count` — number of results returned
+
+### Step 2 Output
+
+`webscraper_fetch(url)` returns full page as Markdown. Use `max_chars` to control size
+(default 5000; use 8000–12000 for deep doc reads).
+
+---
+
+## Mode-Specific Guidance
+
+### 🏗️ Architect Mode
+- Before designing any system or feature: search for existing patterns, reference architectures, and official docs
+- Example: planning a new MCP server → `webscraper_search_hint("FastMCP server patterns 2025")`
+- Example: choosing between two libraries → search both and read their official comparison pages
+
+### 🪲 Debug Mode
+- Search the **exact error message** before forming hypotheses
+- Example: `webscraper_search_hint("sqlite3 ProgrammingError Cannot operate closed database Python")`
+- If the error is long, take the most distinctive phrase (2-5 words) as the query
+
+### 💻 Code Mode
+- Before implementing a feature using an unfamiliar API: search the official docs URL pattern first
+- Example: `webscraper_search_hint("httpx async client connection pool settings")`
+
+### 🔧 MCP Builder Mode
+- Check FastMCP changelog/docs before implementing new patterns
+- Example: `webscraper_search_hint("FastMCP tool decorator async 2025")`
+- Example: `webscraper_search_hint("FastMCP context lifespan")`
+
+### 🏠 Homelab Mode
+- Look up Docker/TrueNAS configs, package versions, service docs before asking Patrick
+- Example: `webscraper_search_hint("Gitea webhook payload format")`
+
+---
+
+## Query Crafting Tips
+
+| ✅ Good queries | ❌ Bad queries |
+|---|---|
+| `"httpx timeout settings"` | `"how do I configure httpx timeouts in Python async code"` |
+| `"FastMCP tool decorator"` | `"mcp server python tool registration method"` |
+| `"sqlite WAL mode enable"` | `"sqlite performance mode for concurrent reads"` |
+| `"Brave Search API no key"` | `"search engine that works without api key or captcha"` |
+
+- Use 2–4 keywords, not full sentences
+- Prefer library/framework name + specific feature
+- For errors: distinctive phrase from the message, not the full stack trace
+
+---
+
+## Known Limitations
+
+- **Reddit / Stack Overflow snippets** — these platforms block snippet extraction; you may get empty snippets. The URL is still valid — fetch it directly if needed.
+- **Brave CSS selector fragility** — Brave uses Svelte-generated class names that change. If `webscraper_search_hint` returns 0 results unexpectedly, the scraper's CSS selectors may need updating. Last verified working: 2026-04-05.
+- **Use sparingly** — one search call per research task to orient; then fetch specific pages. Don't call it in a loop.
+
+---
+
+## Anti-Patterns to Avoid
+
+- ❌ Asking Patrick "what's the FastMCP syntax for X?" before searching
+- ❌ Designing architecture without looking up existing solutions first
+- ❌ Forming a debug hypothesis without searching the error message
+- ❌ Writing code against an API from memory without verifying current docs
+- ❌ Calling `webscraper_search_hint` more than 2-3 times for the same topic (broaden/narrow the query instead)

.roo/skills/gitea-push/SKILL.md

@@ -9,6 +9,7 @@ description: Commits and pushes code to the homelab Gitea server using conventio
 - Finished a homelab change and need to commit + push
 - Finished an MCP server build or update
 - BigMind feature complete
+- Wiki pages were added or updated (always deploy wiki after docs changes)
 
 ## When NOT to use
 - ADP/Paisy work — that goes to the corporate Bitbucket, not homelab Gitea

README.md

@@ -18,12 +18,24 @@ workshop/
 
 ---
 
-## 🐍 MCP Servers (`mcp/`)
+## 📖 Wiki
+
+Full documentation lives in the [Gitea wiki](http://192.168.188.119:30008/pplate/pi_mcps/wiki).
+
+**Wiki source:** [`docs/wiki/pages/`](docs/wiki/pages/) — edit here, deploy with:
+```bash
+./docs/wiki/deploy_wiki.sh
+```
+
+---
+
+## � MCP Servers (`mcp/`)
 
 | Server | Description | Stack |
 |---|---|---|
 | [`mcp/bigmind/`](mcp/bigmind/) | Persistent AI memory — sessions, facts, hypotheses, profile UI | Python, FastMCP, SQLite, Flask |
-| [`mcp/webscraper/`](mcp/webscraper/) | Web scraping — fetch, links, tables, sections, sitemaps | Python, FastMCP, httpx, BeautifulSoup |
+| [`mcp/webscraper/`](mcp/webscraper/) | Web scraping, search — fetch, links, tables, Brave Search | Python, FastMCP, httpx, BeautifulSoup |
+| [`mcp/mcp-image-gen/`](mcp/mcp-image-gen/) | AI image generation — text-to-image via ComfyUI + FLUX.1-schnell | Python, FastMCP, httpx, ComfyUI |
 
 **Run a server:**
 ```bash

docs/wiki/pages/Development-Conventions.md

@@ -145,6 +145,38 @@ Use the `new-mcp-server` Roo skill in MCP Builder mode for full scaffolding:
 3. Roo will load the new-mcp-server skill and scaffold everything
 ```
 
+## Web Research with mcp-webscraper
+
+Before asking Patrick for information about a library, framework, API, or technology — **search first**.
+
+The webscraper MCP server provides `webscraper_search_hint` (Brave Search, no API key, always available) as the entry point for all research tasks. Use the two-step pattern:
+
+```
+Step 1: webscraper_search_hint("topic or error message") → get candidate URLs
+Step 2: webscraper_fetch(best_url)                       → read the full page
+```
+
+### When to search
+
+| Situation | Action |
+|---|---|
+| Need docs for a library or framework | `webscraper_search_hint("library-name official docs")` |
+| Investigating an error or stack trace | `webscraper_search_hint("exact error message language")` |
+| Planning a feature — need design patterns | `webscraper_search_hint("pattern-name best practices")` |
+| Checking latest version / changelog | `webscraper_search_hint("library-name changelog release")` |
+| Looking up API contracts | `webscraper_fetch(official_docs_url)` directly |
+
+### Especially useful in
+
+- **🏗️ Architect mode** — look up patterns and docs *before* designing. Don't design blind.
+- **🪲 Debug mode** — search the exact error message before forming hypotheses.
+- **🔧 MCP Builder mode** — check FastMCP changelog for new patterns before implementing.
+
+### Known caveats
+
+- Reddit and Stack Overflow may return empty snippets (platform blocks)
+- Brave uses Svelte CSS classes that can change — if `webscraper_search_hint` returns 0 results, selectors may need updating (last verified: 2026-04-05)
+
 ## Gitea Repository
 
 Code is hosted at: `http://192.168.188.119:30008/pplate/pi_mcps`

docs/wiki/pages/mcp-webscraper.md

@@ -25,20 +25,70 @@
 - **Search backend:** Brave Search (`search.brave.com`) — works without CAPTCHA
 - **SSL:** Custom cert bundle for Fedora 43 compatibility
 
-## Search Hint Strategy
+---
 
-`webscraper_search_hint` uses Brave Search because:
+## 🔍 Search: The Two-Step Research Pattern
+
+`webscraper_search_hint` is the **entry point for all web research**. The recommended workflow is:
+
+```
+Step 1: webscraper_search_hint("your query") → get candidate URLs + snippets
+Step 2: webscraper_fetch(best_url)           → get full page content
+```
+
+This avoids scraping irrelevant pages and gives you an overview before committing to a deep read.
+
+### Why Brave Search?
+
+`webscraper_search_hint` uses Brave Search (`search.brave.com`) because:
 - ✅ Returns real results without CAPTCHA or consent walls
+- ✅ No API key required — works with plain HTTP GET
+- ✅ Handles special characters (C++, &, %, etc.) via URL encoding
 - ❌ Google blocks plain HTTP with 302 consent redirect
 - ❌ DuckDuckGo blocks with CAPTCHA
 
-Use it sparingly — once per research task — to get oriented before deep-scraping individual pages.
+### Return Value
+
+The tool returns a structured dict:
+
+```json
+{
+  "query": "FastMCP tool decorator",
+  "search_url": "https://search.brave.com/search?q=FastMCP+tool+decorator&source=web",
+  "result_count": 5,
+  "hint": "FastMCP Docs (https://docs.fastmcp.dev): The @mcp.tool() decorator registers a function as... | PyPI FastMCP (https://pypi.org/project/fastmcp/): FastMCP 2.x — modern MCP server framework... | ...",
+  "results": [
+    {
+      "title": "FastMCP Docs",
+      "url": "https://docs.fastmcp.dev",
+      "snippet": "The @mcp.tool() decorator registers a function as an MCP tool..."
+    },
+    ...
+  ]
+}
+```
+
+The `hint` field is a pipe-separated string of `"Title (url): snippet[:120]"` entries — immediately actionable for deciding which URL to fetch next.
+
+### Example: Two-Step Research Flow
 
 ```python
-# Get top 5 results for a query
-webscraper_search_hint("FastMCP tool decorator syntax", max_results=5)
+# Step 1: Orient — what pages exist about this topic?
+result = webscraper_search_hint("httpx async client timeout settings", max_results=5)
+# hint: "HTTPX Docs (https://www.python-httpx.org/...): Configure timeout... | ..."
+
+# Step 2: Deep-dive the most relevant result
+content = webscraper_fetch("https://www.python-httpx.org/advanced/timeouts/", max_chars=8000)
 ```
 
+### Known Limitations
+
+- **Reddit / Stack Overflow snippets** may be empty — these platforms block snippet extraction
+- **Brave CSS selectors** use Svelte-generated class names that may change. If you get 0 results, the scraper's selectors may need updating (last verified: 2026-04-05)
+- **Use sparingly** — once per research task to get oriented, not for every query
+
+---
+
 ## SSL Note — Fedora 43 Comodo Root CA
 
 Fedora 43 is missing the **Comodo AAA Services Root CA** needed for Cloudflare-protected sites. The fix is bundled at [`mcp/webscraper/certs/comodo-aaa-services-root.pem`](../src/branch/main/mcp/webscraper/certs/).
@@ -58,13 +108,16 @@ uv run python src/server.py
 ```bash
 cd mcp/webscraper
 uv run pytest tests/ -v
-# 23/23 tests passing
+# 28/28 tests passing
 ```
 
 ## Usage Examples
 
 ```python
-# Fetch a page as Markdown
+# Step 1: Search — get candidate URLs for a topic
+webscraper_search_hint("FastMCP tool decorator syntax", max_results=5)
+
+# Step 2: Deep-dive the most relevant URL
 webscraper_fetch("https://docs.fastmcp.dev", max_chars=10000)
 
 # Extract all links from Gitea repo
@@ -79,6 +132,6 @@ webscraper_fetch_meta("https://github.com/comfyanonymous/ComfyUI")
 # Fetch specific section by CSS selector
 webscraper_fetch_section("https://docs.python.org", "#content")
 
-# Quick search orientation
-webscraper_search_hint("Gitea wiki git clone", max_results=3)
+# Search with special characters (C++, &, % all work)
+webscraper_search_hint("C++ std::optional usage", max_results=3)
 ```

mcp/webscraper/src/server.py

@@ -3,7 +3,7 @@
 import httpx
 from bs4 import BeautifulSoup
 from html2text import html2text
-from urllib.parse import urljoin
+from urllib.parse import urljoin, quote_plus
 from typing import List, Dict, Tuple
 import re
 import ssl
@@ -275,15 +275,21 @@ def webscraper_search_hint(query: str, max_results: int = 5) -> Dict:
         max_results: Maximum number of results to return (default: 5)
 
     Returns:
-        Dict with 'query', 'results' (list of {title, url, snippet}), 'hint'
+        Dict with 'query', 'search_url', 'results' (list of {title, url, snippet}),
+        'result_count', 'hint'
     """
+    search_url = f"https://search.brave.com/search?q={quote_plus(query)}&source=web"
     try:
-        search_url = f"https://search.brave.com/search?q={query.replace(' ', '+')}&source=web"
         _, soup = _fetch_page(search_url)
 
         results = []
-        # Brave Search result cards: each <a> with class snippet contains title + description
-        for card in soup.select('.snippet')[:max_results]:
+        seen_urls: set = set()
+
+        # Brave Search result cards: each div.snippet contains title, URL, description
+        for card in soup.select('.snippet'):
+            if len(results) >= max_results:
+                break
+
             title_el = card.select_one('.snippet-title')
             url_el = card.select_one('a')
             desc_el = card.select_one('.snippet-description')
@@ -292,20 +298,48 @@ def webscraper_search_hint(query: str, max_results: int = 5) -> Dict:
             url = url_el['href'] if url_el and url_el.get('href') else ""
             snippet = desc_el.get_text(strip=True) if desc_el else ""
 
-            if url and url.startswith('http'):
-                results.append({"title": title, "url": url, "snippet": snippet})
+            # Filter: must have a valid http(s) URL
+            if not url or not url.startswith('http'):
+                continue
 
-        hint = "; ".join(
-            f"{r['title']}: {r['url']}" for r in results
-        ) if results else "No results found"
+            # Filter: skip results with no useful content at all
+            if not title and not snippet:
+                continue
+
+            # Deduplicate by URL
+            if url in seen_urls:
+                continue
+            seen_urls.add(url)
+
+            results.append({"title": title, "url": url, "snippet": snippet})
+
+        # Richer hint: title + url + first 120 chars of snippet for AI context
+        if results:
+            hint_parts = []
+            for r in results:
+                part = f"{r['title']} ({r['url']})"
+                if r['snippet']:
+                    part += f": {r['snippet'][:120]}"
+                hint_parts.append(part)
+            hint = " | ".join(hint_parts)
+        else:
+            hint = "No results found"
 
         return {
             "query": query,
+            "search_url": search_url,
             "results": results,
+            "result_count": len(results),
             "hint": hint,
         }
     except (httpx.RequestError, httpx.HTTPStatusError) as e:
-        return {"query": query, "results": [], "hint": f"Error: {str(e)}"}
+        return {
+            "query": query,
+            "search_url": search_url,
+            "results": [],
+            "result_count": 0,
+            "hint": f"Error: {str(e)}",
+        }
 
 
 if __name__ == "__main__":

mcp/webscraper/tests/test_server.py

@@ -234,18 +234,92 @@ def mock_brave_response():
     return mock_resp
 
 
+@pytest.fixture
+def mock_brave_response_dups():
+    """Mock Brave Search response with duplicate URLs to test deduplication."""
+    mock_resp = MagicMock()
+    mock_resp.status_code = 200
+    mock_resp.text = """
+    <html><body>
+        <div class="snippet">
+            <a href="https://example.com/dup">Dup Result A</a>
+            <div class="snippet-title">Dup Result A</div>
+            <div class="snippet-description">First occurrence.</div>
+        </div>
+        <div class="snippet">
+            <a href="https://example.com/dup">Dup Result B</a>
+            <div class="snippet-title">Dup Result B</div>
+            <div class="snippet-description">Second occurrence — same URL.</div>
+        </div>
+        <div class="snippet">
+            <a href="https://example.com/unique">Unique Result</a>
+            <div class="snippet-title">Unique Result</div>
+            <div class="snippet-description">Only once.</div>
+        </div>
+    </body></html>
+    """
+    mock_resp.headers = {"content-type": "text/html"}
+    return mock_resp
+
+
+@pytest.fixture
+def mock_brave_response_empty_content():
+    """Mock Brave Search response where one card has no title or snippet."""
+    mock_resp = MagicMock()
+    mock_resp.status_code = 200
+    mock_resp.text = """
+    <html><body>
+        <div class="snippet">
+            <a href="https://example.com/ghost"></a>
+            <div class="snippet-title"></div>
+            <div class="snippet-description"></div>
+        </div>
+        <div class="snippet">
+            <a href="https://example.com/real">Real Result</a>
+            <div class="snippet-title">Real Result</div>
+            <div class="snippet-description">Has content.</div>
+        </div>
+    </body></html>
+    """
+    mock_resp.headers = {"content-type": "text/html"}
+    return mock_resp
+
+
 @patch('httpx.get')
 def test_webscraper_search_hint_returns_structure(mock_get, mock_brave_response):
-    """Test that search hint returns correct dict structure."""
+    """Test that search hint returns all required dict fields."""
     mock_get.return_value = mock_brave_response
     result = webscraper_search_hint("Feynman electric field")
     assert isinstance(result, dict)
     assert "query" in result
+    assert "search_url" in result
     assert "results" in result
+    assert "result_count" in result
     assert "hint" in result
     assert result["query"] == "Feynman electric field"
 
 
+@patch('httpx.get')
+def test_webscraper_search_hint_search_url_encoded(mock_get, mock_brave_response):
+    """Test that search_url uses proper URL encoding (quote_plus, not str.replace)."""
+    mock_get.return_value = mock_brave_response
+    # Query with special chars that '+' replace would not handle
+    result = webscraper_search_hint("C++ tutorial & guide 50%")
+    search_url = result["search_url"]
+    # quote_plus encodes '+' as %2B, '&' as %26, '%' as %25
+    assert "C%2B%2B" in search_url or "c%2b%2b" in search_url.lower()
+    assert "%26" in search_url
+    assert "%25" in search_url
+
+
+@patch('httpx.get')
+def test_webscraper_search_hint_result_count(mock_get, mock_brave_response):
+    """Test that result_count matches the number of results returned."""
+    mock_get.return_value = mock_brave_response
+    result = webscraper_search_hint("Feynman electric field")
+    assert result["result_count"] == len(result["results"])
+
+
 @patch('httpx.get')
 def test_webscraper_search_hint_filters_non_http(mock_get, mock_brave_response):
     """Test that javascript: URLs are excluded from results."""
@@ -262,25 +336,64 @@ def test_webscraper_search_hint_max_results(mock_get, mock_brave_response):
     mock_get.return_value = mock_brave_response
     result = webscraper_search_hint("Feynman electric field", max_results=1)
     assert len(result["results"]) <= 1
+    assert result["result_count"] <= 1
+
+
+@patch('httpx.get')
+def test_webscraper_search_hint_deduplicates_urls(mock_get, mock_brave_response_dups):
+    """Test that duplicate URLs are deduplicated — only first occurrence kept."""
+    mock_get.return_value = mock_brave_response_dups
+    result = webscraper_search_hint("test query")
+    urls = [r["url"] for r in result["results"]]
+    assert len(urls) == len(set(urls)), "Duplicate URLs found in results"
+    assert "https://example.com/dup" in urls
+    assert "https://example.com/unique" in urls
+    assert len(urls) == 2  # dup appears once, unique once
+
+
+@patch('httpx.get')
+def test_webscraper_search_hint_filters_empty_content(mock_get, mock_brave_response_empty_content):
+    """Test that cards with no title AND no snippet are excluded."""
+    mock_get.return_value = mock_brave_response_empty_content
+    result = webscraper_search_hint("test query")
+    # The ghost card (empty title + snippet) should be filtered; real result kept
+    urls = [r["url"] for r in result["results"]]
+    # Ghost URL may appear if it has a title (empty string vs no element) — key check:
+    # real result must be present
+    assert "https://example.com/real" in urls
 
 
 @patch('httpx.get')
 def test_webscraper_search_hint_error(mock_get):
-    """Test error handling in search hint."""
+    """Test error handling in search hint — returns all required fields."""
     mock_get.side_effect = httpx.RequestError("Connection failed")
     result = webscraper_search_hint("something")
     assert result["results"] == []
+    assert result["result_count"] == 0
     assert "Error" in result["hint"]
+    assert "search_url" in result
+    assert "query" in result
 
 
 @patch('httpx.get')
-def test_webscraper_search_hint_hint_string(mock_get, mock_brave_response):
-    """Test that hint string is non-empty when results exist."""
+def test_webscraper_search_hint_hint_includes_snippet(mock_get, mock_brave_response):
+    """Test that the hint string includes snippet content, not just title+url."""
     mock_get.return_value = mock_brave_response
     result = webscraper_search_hint("Feynman electric field")
-    # hint should summarise results
-    assert len(result["hint"]) > 0
+    # hint should contain snippet text
+    assert "electric field" in result["hint"].lower()
     assert "No results found" not in result["hint"]
+    assert len(result["hint"]) > 0
 
 
-# Total: 23 tests covering all tools and edge cases
+@patch('httpx.get')
+def test_webscraper_search_hint_hint_format(mock_get, mock_brave_response):
+    """Test that hint uses pipe-separated format with URL in parens."""
+    mock_get.return_value = mock_brave_response
+    result = webscraper_search_hint("Feynman electric field")
+    # Format: "Title (url): snippet | Title2 (url2): snippet2"
+    assert "(" in result["hint"]
+    assert ")" in result["hint"]
+
+
+# Total: 31 tests covering all tools and edge cases