I Built a Desktop AI Assistant That Controls Your Computer — Here's How

TL;DR: I built Yaldabaoth — a desktop AI assistant that doesn't just answer questions. It reads your screen, runs PowerShell commands, clicks buttons, types text, and automates your entire workflow. No cloud dependency. No API calls for automation. Just Python, Rust, and raw OS control.

The Problem

In 2025, "AI assistants" meant one of two things:

1. Chatbots — A text box where you type and an LLM talks back
2. API wrappers — Tools that chain API calls together but have zero ability to touch the actual operating system

Neither of these actually helps you do things on your computer. Want to open an application, take a screenshot, parse a PDF, run a PowerShell script, and compile a report? Good luck chaining that through a chat interface.

I wanted something different. An assistant that sits on your desktop, sees what you see, and acts on your behalf — like having an engineer sitting next to you who can operate any part of the system.

So I built Yaldabaoth.

The Architecture

Yaldabaoth is a four-layer system:

Layer 1: The Shell — Tauri + React

Instead of Electron (which would have added 150+ MB to the binary), I used Tauri — a Rust-based framework that wraps a webview frontend in a native shell. The UI is React with a glassmorphism design.

Why Tauri:

• Binary size: ~5 MB vs Electron's ~150 MB
• Native performance for intensive operations
• Direct Rust system access when needed

Layer 2: The Orchestrator — Python Backend

The Rust shell communicates with a Python backend that handles all the heavy lifting. Communication is via stdin/stdout JSON protocol — lightweight, no HTTP server needed.

The orchestrator manages:

• Command routing (voice/text → appropriate handler)
• State persistence between commands
• Multi-step task chaining
• Personality profile switching (Professional vs. creative modes)

Layer 3: The Automation Engine — Win32 API + PowerShell

This is where the magic happens. The Python backend has direct access to the Windows OS through:

• pywinauto — Native Win32 API control for clicking, typing, window management
• PowerShell subprocess — OS-level commands (service control, registry edits, file operations)
• WMI — System information queries (processes, hardware, network)

Layer 4: The Perception System — OCR + Screen Parsing

Screen parsing runs in a separate thread to keep the UI responsive. It uses:

• OCR-based text extraction — Screenshots → text → action decisions
• Multi-threaded processing — One thread for screen capture + OCR, another for command execution, a third for UI responsiveness
• Chained automation — Click → wait for UI update → re-scan screen → next action

The Hard Parts

Threading Nightmares

Screen parsing is slow. OCR a screenshot, parse the text, decide what to do — you're looking at 500ms to 2 seconds per cycle. If you do this on the main thread, your entire app freezes.

The solution was a producer-consumer architecture:

• Thread 1: Screen capture → OCR → queue the parsed text
• Thread 2: Command executor — reads from queue, takes action
• Thread 3: Main UI thread — stays responsive

┌─────────┐    ┌──────────┐    ┌──────────┐
│ Screen  │───>│ Queue     │───>│ Command  │
│ Capture │    │ (JSON)   │    │ Executor │
└─────────┘    └──────────┘    └──────────┘
      │                              │
      v                              v
 ┌─────────┐                   ┌──────────┐
 │ OCR     │                   │ Win32 API│
 │ Parser  │                   │/PowerShell│
 └─────────┘                   └──────────┘

Python's threading.Queue with daemon=True threads was sufficient — no need for multiprocessing or async for this use case.

Voice-First vs. Text-First UX

I wanted a Push-to-Talk interface (F10 key) so you could speak commands naturally. But speech recognition introduces latency and errors. The compromise:

• Voice input preferred for simple commands ("Open Chrome", "Check CPU usage")
• Text fallback for complex multi-step sequences
• The orchestrator normalizes both into the same command pipeline

Rust ↔ Python Bridge

Tauri apps expect Rust backends. Yaldabaoth needs Python. Bridging them without adding a web server was tricky.

The solution: stdin/stdout JSON-RPC. The Rust shell spawns the Python process and communicates via JSON messages on stdin/stdout. No sockets, no HTTP, no dependency on a running server. The Python process lives as long as the app is open.

// Rust side — minimal example
let python = Command::new("python")
    .arg("backend/main.py")
    .stdin(Stdio::piped())
    .stdout(Stdio::piped())
    .spawn()?;

// Send command
let cmd = r#"{"action": "click", "target": "Chrome"}"#;
python.stdin.as_ref().unwrap().write_all(cmd.as_bytes())?;

// Read response
let mut response = String::new();
python.stdout.as_ref().unwrap().read_to_string(&mut response)?;

What I Learned

1. Desktop automation is harder than cloud automation. Cloud APIs are designed to be called programmatically. Desktop UIs are designed for humans. Parsing a rendered UI and making decisions from it is fundamentally different from calling an API endpoint.

2. Threading early, threading often. I rebuilt the threading model three times. The first version was single-threaded and froze constantly. The second over-engineered with multiprocessing. The third — simple Queue-based threading — was just right.

3. Computer use was already possible in 2025. Before "computer use" became a buzzword in 2026, a Python script + OCR + Win32 API was all you needed. The novelty isn't the technology — it's wiring it together with a voice-first, responsive UI that feels like an assistant, not a script.

The Tech Stack

What's Next

• Cross-platform support — Currently Windows-only due to Win32 API dependency. Linux adaptation via X11/Wayland is on the roadmap.
• Better screen parsing — Using vision models directly instead of OCR for richer UI understanding.
• Plugin system — Let users write custom automation modules.

Repo

The full source code is on GitHub: github.com/HENI-MOHAMED/Yaldabaoth

Built with Tauri, React, Python, Rust, and more coffee than I'd like to admit.