Skip to content
Synoec

Web search and scraping

Two independent services: Search (SearXNG) finds sources and Scrape (Scrapling) reads chosen pages. Each is off, Managed local (Synoec downloads it and starts it when needed) or Configured remote (your own deployment). Managed local runs on macOS, Linux and Windows x64. Retrieved text is evidence, never instructions. For interactive pages use Agora.

Tool Use
WebSearch Ranked links and snippets
WebScrape Text or CSS fields from pages, URL batches or bounded same-host crawls
WebFetch A known plain-text or API resource; oversized bodies return a marked prefix
WebAvailability Which web services are on and installed, without starting anything

Set up. Open Settings → Web & Network. Each card (Search, Scrape) shows Installed version, Runtime size and Health. Under Where it runs pick Managed local and choose Download & Enable (private Python and browser runtimes, no Docker; Repair & Enable after an interrupted setup), or pick Configured remote, enter the Endpoint and an optional Bearer token environment variable name, then Enable or Save changes. Diagnose, Repair installation and Disable sit on the card; Refresh at the top re-reads both without starting anything. A failed setup keeps your previous configuration.

Terminal window
synoec web enable search --mode local
synoec web enable scrape --mode local
synoec web enable all --mode remote --endpoint https://web-tools.internal --bearer-env WEB_TOOLS_TOKEN
synoec web status all
synoec web doctor search
synoec web repair scrape
synoec web disable all

--bearer-env NAME stores the variable name, not the secret. A remote outage never starts local services. If Search is off, the agent asks you to enable it or answers without browsing; it does not guess URLs.

How search stays on topic: managed search asks Bing, DuckDuckGo, Google, Wikipedia, Yahoo and Mwmbl at once. Engines often refuse automated requests; results name any that did, such as google (access denied). Results that mention none of the query’s subject words are dropped, duplicates count once, and when nothing relevant is left the agent is told to rephrase or report that discovery failed. An HTTP 429 or engine suspension is reported as an upstream limit, not a login failure.

How scraping recovers: redirects from a named page are followed (twitter.com reaches x.com), text comes from the main content, and a page that needs JavaScript is rendered once in a browser automatically. If local Scrape is on but not yet installed, plain page reads use a built-in reader; rendering, crawls and CSS fields need Scrape installed. A failed page returns a reason: not_found, access_denied (bot checks are never bypassed), rate_limited, unsupported_media (download PDFs with the browser, then read the file), timeout, http_status or network_failure.

To check real queries and pages, run bun packages/native-bridge/scripts/web-search-live-check.ts or web-scrape-live-check.ts from the repository.

Run your own service. From the repository root:

Terminal window
docker compose -f deploy/web-tools/compose.yml build
docker compose -f deploy/web-tools/compose.yml up -d

Supply unique SEARXNG_SECRET, SYNOEC_SEARXNG_TOKEN and SYNOEC_SCRAPLING_TOKEN from a secret store. Compose binds to loopback; set SYNOEC_WEB_BIND_ADDRESS to expose it on a private interface such as your NetBird network. The search gateway serves /config and /search, the scrape service /health and /v1/scrape. Configure matching bearer variables, then run synoec web doctor for both. See deployment sources and the rich-tool contract.