Skip to content
Synoec

Local AI

Manage the Ollama you run and the Bonsai 2 Synoec hosts with NInfer, on one page.

Local AI is a sidebar page under Marketplace. It appears only when it has something to manage: the workspace has an Ollama Connection, or, on Windows, Bonsai 2 is set up or this PC’s NVIDIA GPU can run it. It has two tabs, each shown under the same rule: Ollama and Synoec. Neither changes a Profile or a chat’s model.

Limits

Section titled “Limits”
  • Measured once, on the build VM’s RTX 3090 Ti (driver 617.42, NInfer 2026-11-03.1): setup picked 262,144 tokens of rk8v4; the first load took 2 m 30 s (route calibration, done once per card) and used 16.4 GB of the card; a short reply answered in 0.2 s; the model left the card after five idle minutes (860 MiB in use) and came back for the next request in 6.7 s. A later run on 2026-11-08 saw a reload take over three minutes, which is why a slow first byte is now reported as slowness rather than an unreachable provider. Other cards have not been measured.
  • 8 GB GPUs cannot host it, and smaller weights or CPU fallback are never substituted.
  • Several graphics cards: Bonsai and a local Ollama take turns on the GPU even on a PC with more than one card, where Ollama could run on another card. Synoec does not know which card Ollama uses.
  • Context use on small local models: Synoec’s own instructions and tool list take tens of thousands of tokens of every request, and are the same on every model. On a 40,960-token model such as qwen3:8b that leaves little room for the conversation.
Section titled “Related”

Decisions behind this page: 0074, 0235, 0240, 0241, 0242, 0244, 0286 — see the decisions index.