Troubleshooting
Blocked pages, empty extractors, OCR, OAuth, timeouts — symptom → cause → fix.
Symptom → likely cause → what to do. Every item here is something that happened during testing.
Pages and sessions#
| Symptom | Cause | Fix |
|---|---|---|
challenge: captcha / cloudflare / login on a session or scrape | The site challenged a headless browser or your IP | CrawlGenie does not solve challenges. Take over in the live view, use a profile with a saved login, or route through a residential proxy. Turn on signed agent so sites that honour Web Bot Auth can recognise you. |
| Extractor returns 0 rows with "Blocked" | Same as above, or the site needs a proxy (needs: proxy in the catalogue) | Run with session.proxy, or pick an API-backed extractor for the same data |
| Extractor returns 0 rows, no challenge | The page structure changed or the search had no results | Open the URL in a session and look; use css_extract or auto_list as a fallback; report the extractor id |
navigation_blocked … private network | SSRF guard | Set policy.allowPrivateNetwork: true for that session, or VESSEL_SSRF_GUARD=false on a trusted network |
Session ended with idle_timeout | No action for the idle window | Raise policy.idleTimeoutSec or the default in Settings |
| Snapshot is huge | Page has thousands of interactive nodes | Use mode: interactive, maxTokens, or q= on text; prefer web_answer when you only need to read |
| Approval never resolves | Nobody decided | Approvals page, webhook approval.pending, or set approvals.timeoutSec |
Documents and OCR#
| Symptom | Cause | Fix |
|---|---|---|
| PDF returns almost no text | Scanned PDF | ocr: auto already triggers on sparse text; use ocr: force if the text layer is garbage |
| OCR says "language data unreachable" | First run needs to download Tesseract data | Allow outbound HTTPS once, or pre-seed data/tesseract |
| Table columns are merged | Cells are closer than the column threshold | Use granularity: sentence with q, or export the page range and post-process; report the PDF |
| Google Doc returns HTML or 401 | The document is not shared "anyone with the link" | Share it, or export manually and upload |
| YouTube: "no transcript" | Video has no caption track | Nothing to read without a speech model |
Connectors, MCP and auth#
| Symptom | Cause | Fix |
|---|---|---|
| ChatGPT / Claude cannot connect | CrawlGenie is not reachable over public HTTPS, or VESSEL_PUBLIC_URL differs from the URL you entered | Set VESSEL_PUBLIC_URL to the exact public origin and restart; check /.well-known/oauth-authorization-server |
| Consent page says "cross-site request refused" | CSRF guard | Open the consent page from the platform's flow, not from a saved link |
401 unauthorized from /v1 | Missing or wrong key | Authorization: Bearer <key>; create keys under API keys |
| Savings page shows no connector line | No session has been created through that connector yet | Ask the platform to use a CrawlGenie tool once; the channel appears as connector:<name> |
Watches and schedules#
| Symptom | Cause | Fix |
|---|---|---|
| Watch never runs | Server was down at cron time, or hosted on a platform that sleeps | Use POST /v1/tick from an external scheduler (GitHub Actions, Cloud Scheduler) |
| Every run reports "changed" | The page contains a clock, counters or rotating content | Use an extractor with a key (rows) instead of change_monitor, or narrow the selector |
| Webhook not received | URL unreachable from the server, or no webhookUrl | Settings → webhook URL; check the server log for webhook errors |
Knowledge and help#
| Symptom | Cause | Fix |
|---|---|---|
| "Nothing matches" though the text is there | Different words than the source | Use the source's vocabulary, or add a short glossary document |
| Help assistant answers from the wrong page | Several pages share the words | Click the link to the doc; ask with the feature name (q, pages, watch) |
| Docs page says the folder is missing | The server runs outside the repo and the image was built without docs | Set VESSEL_DOCS_DIR, or rebuild the image (docs are copied in) |
Where to look#
- Session page → Timeline: every step with a screenshot and the policy decisions.
GET /v1/sessions/{id}/eventsandGET /v1/statsfor scripts.- Server log (JSON):
level: 50lines are unhandled errors; include them in a bug report with your version.
Section: Run · This page as markdown · Questions? hello@crawlgenie.com