The Brain v1.0 — building open-source workflow orchestration the boring way

Originally published on mihaibuilds.com. Cross-posting here because dev.to is where I find a lot of this kind of work myself.

Memory Vault shipped five weeks ago — the substrate for the planned compounding stack. Memory is one half of "your AI assistant should know what you've done." The other half is "your tools should run when they should, against the right state, with the history you can audit." Cron lines on a server you half-remember and a Python file glued to a webhook glued to an LLM is not that. The Brain is the runtime underneath: Python-defined workflows, every run persisted, owned by you.

The Brain is open-source, self-hosted, MIT-licensed. A Python file defines the workflow. Postgres holds every run. Four trigger types fire it (manual, cron, webhook, file). Four step types execute inside it (shell, LLM, Memory Vault REST, MCP). The whole thing is one multi-arch Docker image. Today it crosses the line from build-in-public project to v1.0 stable release.

What The Brain is

A workflow orchestrator with persistent run history. You define a workflow as a Python file — a sequence of named steps. The Brain runs it on whichever of the four triggers you registered (a CLI invocation, a cron expression, an HMAC-signed webhook, or a filesystem change), persists every step's result to a single Postgres database, and stops on the first failure with a clear error. brain history shows what ran. brain show <run-id> shows the per-step detail. brain diagnose produces a redacted zip ready to attach to a bug report.

It runs entirely on your machine. No SaaS account. No telemetry. No cloud lock-in. docker compose --profile api up -d and it's running. The CLI alone works against a Dockerized Postgres if you don't need the API or the watcher daemon.

The biggest thing about The Brain is what it deliberately doesn't do: it doesn't decide anything on its own. Workflows are scripted Python. The LLM step just transforms text. If a workflow wants "the LLM picks the next tool to call," you write that workflow — but the workflow file is doing the picking, not The Brain. The Brain is a workflow orchestrator, not an agent runtime.

What v1.0 actually does

Four step types — ShellStep runs subprocess commands with stdout capture and exit-code semantics; LLMStep calls an OpenAI-compatible chat API with per-step provider/model/key/timeout overrides; MemoryVaultStep calls Memory Vault's REST API for the friction-free case; McpToolStep calls any MCP tool over stdio with per-step spawn lifecycle.
Four trigger types — manual (CLI), cron (scheduler daemon), webhook (HTTP POST with HMAC signature, verified constant-time), file (filesystem watcher with debounced events).
Substitution between steps — {previous.step-name} references prior step output; {trigger.body} / {trigger.path} / {trigger.headers.X} reference trigger payload data. Textual substitution, not eval'd Python — str.format shape, strict-by-default errors when a reference doesn't resolve.
Three processes, one database — the CLI for one-shot runs, brain serve for the HTTP API + webhook intake, brain watch for cron and file triggers. Compose profiles let you run any subset.
Structured logging with run_id binding — bind_run_id() attaches the run UUID to every log line emitted during a run. LOG_FORMAT=keyvalue for human-readable console output, LOG_FORMAT=json for log aggregation pipelines.
Redacted diagnostic bundle — brain diagnose writes a timestamped zip with recent logs, environment, OS info, Brain version, and Docker container state. The redaction model is allow-list, not blocklist: nine env vars are recorded with values; four (DB_PASSWORD, LLM_API_KEY, MEMORY_VAULT_TOKEN, THE_BRAIN_API_TOKEN) are presence-only — name appears, value never appears in the bundle.
One-command Docker, multi-arch — linux/amd64 and linux/arm64 images published to ghcr.io/mihaibuilds/the-brain. Compose profiles for api and watcher. Derive your own image when you need to add MCP servers or extra binaries.
MIT-licensed, self-hosted — your workflows, your data, your hardware. The whole thing is yours.
363 tests passing — pytest with a real Postgres service container, no mocks at integration boundaries.

Architectural decisions worth naming

Postgres is the only state store. No Redis. No queue. No second datastore. Run history, webhook secrets, and scheduler state all live in the same Postgres database, with LISTEN/NOTIFY for cross-process wakeup and plain SQL polling for the rest. You already know how to back up Postgres. You already know how to monitor it. Adding a queue or a Redis would double the backup story and the operational mental model for no win at the scale a self-hosted workflow runner actually runs at. When that stops being true, the migration path is sane. Until then, one database is the right answer.

Three processes, not one. The runner is synchronous per-run, the HTTP API is async per-request, the watcher is a long-running event loop. Splitting them means a crashing scheduler doesn't take the API down, and operators can run only the parts they need via Compose profiles. CLI-only users skip the daemons entirely. API-only users skip the watcher. The full stack runs all three. Process boundaries are cheaper than recovery procedures.

Per-step subprocess spawn for shell and MCP. A crashing shell command or a misbehaving MCP server should not take the runner down. ShellStep always spawns a fresh subprocess per step. McpToolStep spawns a fresh MCP server, runs the protocol handshake, makes one tool call, and tears it down at step end. No shared client. No pooling. The fork/exec cost is in the noise for the workflow shapes The Brain targets, and the isolation is real — if an MCP server crashes mid-call, only that one step fails and the next step gets a brand-new subprocess.

Workflow files are trusted Python, not a YAML DSL. Every YAML workflow language ends up needing escape hatches that turn back into code. Workflow-as-Python means full editor support, full type checking, zero new syntax to learn — at the cost of treating workflow files as part of your trust boundary, same as any other file in your repo. The threat model in SECURITY.md says so up front rather than hiding it behind a "do not run untrusted workflows" footnote.

The first time the ecosystem composed end-to-end

Memory Vault has been live since early May. The Brain has been under construction since mid-May. I've been calling them "the ecosystem" the whole time, but they were two completely separate products living in two completely separate places — separate Postgres databases, separate codebases, separate Docker images. They had never actually worked together end-to-end.

v1.0 of The Brain is the first time the two compose in production shape. The pattern is what I call horizontal substrate, vertical products: Memory Vault is the substrate that stores and retrieves; The Brain is the engine that runs workflows; both run independently, both compose together. A workflow on The Brain can ask Memory Vault for memories over MCP, pipe those memories into an LLM step that summarizes them, write the summary to a file. Real Postgres on both sides. Real MCP stdio transport. Real LLM. Real file written.

The integration path is the derive-pattern: you write a Dockerfile that derives FROM ghcr.io/mihaibuilds/the-brain:1.0, installs Memory Vault into the image, and the workflow step's server_command spawns Memory Vault's MCP server inside the same container as a per-step subprocess. The Brain ships zero MCP servers in the stock image — that's deliberate. The Brain stays small. Memory Vault stays independent. The composition is opt-in and explicit. The same shape works for any MCP server: GitHub's, Sentry's, your own.

This is the moment "the ecosystem" stops being something I write on a roadmap and becomes a system that genuinely exists. Two products. Two databases. One Docker network. Composing.

The version-string story

I tagged v1.0.0 from main after the security audit pass. The release workflow fired, built the multi-arch image, pushed it to ghcr.io, and auto-published the GitHub Release from my annotated tag message. Two minutes twenty-nine seconds, end to end. Clean.

Then I ran the smoke test. docker pull ghcr.io/mihaibuilds/the-brain:1.0.0 succeeded. Multi-arch manifest verified — linux/amd64 and linux/arm64 both present. I ran brain --version inside the published image to confirm the version string matched the tag.

It returned brain, version 0.0.1.

I never bumped pyproject.toml from 0.0.1. The image tag said 1.0.0. The CLI inside the image said 0.0.1. The release was published, immutable, and visibly inconsistent on every fresh install.

The fix was four lines. Three options were on the table: hotfix to v1.0.1, delete the published tag and re-tag (which would break anyone who'd already pulled :1.0.0), or leave it for the natural course of a v1.0.x. The right call was the cheap one — hotfix. Bump pyproject to 1.0.1 (skipping 1.0.0 because that tag is published and immutable), open a one-file PR, watch CI go green, merge, tag v1.0.1, push. The release workflow re-fired and republished. Five minutes later, brain --version inside :1.0.1 and :latest both returned brain, version 1.0.1. The digest is sha256:41d0becdf7d036….

The lesson isn't about version strings. It's that automated release infrastructure makes hotfix cycles cheap, and "cheap to do correctly the second time" is what gives you permission to do v1.0.0 with confidence rather than paralysis. The pre-tag checklist now has a pyproject.version line. Future ecosystem product launches will hit this gate before the tag is pushed. The hotfix lesson costs five minutes of CI; the lesson is permanent.

The discipline that built the v1.0 ship survived its own first encounter with reality — by treating a real release-engineering bug like a real bug, not by quietly editing history.

What v1.0 doesn't do, on purpose

No retries on step failure. First step failure halts the workflow. The persisted run row shows exactly where it stopped and why. Retry-on-failure with backoff is a v1.x candidate when real users ask for it, not a v1.0 default that hides bugs.

No catch-up on missed cron windows. If The Brain is down when a cron expression matches, that window is skipped — not deferred and replayed. Distributed time-shifted scheduling is its own product; The Brain assumes the host is online when the cron fires.

No tools/list discovery for MCP servers. Workflow authors know the tool name and argument shape in advance, like they know which shell commands they're calling. Discovery is an interactive-IDE problem, not a workflow-orchestrator problem.

No MCP over HTTP transport. Stdio only in v1.0. HTTP transport is a v1.x candidate if real demand surfaces.

No LLM-driven tool orchestration. LLMStep is chat-completion only. If a workflow wants "the LLM picks the MCP tool to call," that's a two-step pattern: the LLM step produces a tool name, the substitution pipes it into a downstream McpToolStep. The workflow is the agent. The Brain is the runtime that runs the workflow.

No bundled MCP servers in the stock image. The base image is intentionally minimal. Derive-your-own-image is the documented composition path.

No multi-user. v1.0 is single-tenant. The single THE_BRAIN_API_TOKEN env var is the auth model. Multi-user with workspace isolation is part of the planned PRO tier.

No visual workflow builder. Workflows are Python.

These are deliberate trade-offs. Honest gaps documented up front build more trust than feature bullets that fall apart when someone actually tries them.

The open-core model

The Brain is and will always be MIT-licensed. The whole thing — the runner, the four step types, the four trigger types, the HTTP API, the CLI, the scheduler daemon, the watcher daemon, the structured logging, the diagnose bundle, the multi-arch Docker images. You can run it on your machine. You can fork it. You can use it inside a commercial product. The free tier is the full runtime, not a crippled demo of a paid tier.

A paid PRO tier is planned: multi-user with workspace isolation, a hosted scheduler for people who don't want to run their own watcher daemon, a secrets vault, conflict resolution for shared workflow repos, additional integrations. The PRO tier is genuinely paid operational features — what teams running shared workflow infrastructure actually need, not what a solo developer on a laptop strictly needs. v1.x stays free forever.

Try it

docker pull ghcr.io/mihaibuilds/the-brain:latest

git clone https://github.com/MihaiBuilds/the-brain
cd the-brain
cp .env.example .env
docker compose --profile api up -d