timescale.info#163 · Compile plugin scripts only in a sandbox; plugins read their own metadata
- timeline-app
- pull request
- plugins
Closes #157. The server never runs plugin code. It now compiles plugin scripts only in a sandbox, and each plugin reads its own metadata. ## Compiling in a sandbox **Before**, `validatePluginManifest` compiled the script in the request's own thread whenever a manifest was validated: - every server-side `validateDocument`; - once per changed moment of a patch, since each change was validated with the timeline's settings; - catalogue reads; - comparison query plugins. **Now**, manifest validation checks only shape and size. The server compiles a script only through `server/plugin-sandbox.mjs`: - **Workers:** worker threads load only the compiler, `dist/plugin-compile-worker.mjs`, a 58 KB bundle built from `src/plugin-compile-worker.mjs`. - **Isolation:** a worker gets an empty environment and only the source string. - **Limits:** 64 MiB of heap, a 4 MiB stack and 2 s per script. A worker past a limit is terminated and replaced, and the request's thread is unaffected. - **Pool and cache:** a pool of two workers, with results cached by the source's SHA-256. Refusals for exceeding a limit aren't cached, so a busy moment doesn't refuse a script for good. **When it compiles:** - **Timelines:** `store.create`, and `store.save` for whole documents and patches. Only scripts that are new count: not a built-in plugin's, and not one the timeline already has. Most saves compile nothing. - **Pull requests:** create and update with a document, against the upstream's plugins. - **Publications:** publishing a plugin or a suite. - **Operator libraries:** a `PLUGIN_LIBRARY` file, at startup. Catalogue reads and comparison query plugins no longer compile anything: they store nothing and run nothing on the server. The compiler is already bounded (16 KiB of source, 4,096 tokens, 1,024 nodes, nesting 32 deep), so no real script reaches the sandbox's limits. The test uses a hostile stand-in worker to show the limits hold. The browser compiles when you author, import or install a plugin, so errors show early. It also compiles before running a script, as before. ## Plugins read their own metadata - **Validation removed:** `validateStackMetadata` is gone, from `validateDocument` and from the moment editor. The core still checks only its own fields (title, notes, tags) and the generic limits. - **Lenient stack reading:** `readStack` and `stackEntries` read any value and never throw. A readable entry is exactly an object with a unique ASCII ID and metadata, a text title and notes, and no stack of its own. Anything else is counted as unreadable. - **Details panel:** it shows the readable entries. A stack with unreadable entries is shown read-only with a note pointing to **Additional metadata**, so editing never silently drops them. "Add entry to stack" does nothing on such a stack. - **Stack editor:** it still refuses what it can't read (`stackEntry`). - **Timeline markers and comparison rows:** they use the same reader. - **Other field kinds:** they already read leniently. A test now covers each kind with wrong-shaped values. - **Log importer:** it sends plugin keys as given. ## Plugin changes save in place The plugin condition is gone from `incremental` (`server/incremental.mjs`). Changing the installed plugins is a settings change: no moment is revalidated and no index is rebuilt. Overviews project fields per query from the current plugins, so nothing goes stale. ## Tests - **`test/plugin-sandbox.test.mjs`** (new): - A real script compiles; a failing one is refused with the compiler's message; repeats are cached. - With `test/hostile-compiler.mjs`, a worker that never answers is stopped after 300 ms while the event loop keeps running. - One that allocates without end is stopped at its memory limit. Both are replaced and the next script compiles. - Workers are pooled, and a worker sees an empty environment. - **`test/plugins.test.mjs`:** - Every built-in plugin's script compiles (nothing compiles them on load now). - Every field kind of every built-in pl…