What if your home could understand you — without sending a single word to the cloud?
That question started this project. I wanted to control my smart home with voice commands in Hungarian — a language that sits far outside the English-centric comfort zone of most voice assistants. I wanted context awareness: the system should know which lights are already on, what time of day it is. And I wanted it to be private: no audio recordings uploaded to someone else's servers, no device state telemetry leaving my network.
What I did not expect was that the journey from cloud to local AI would end with my local setup outperforming the cloud version. This is the full story — with the raw numbers to prove it.
The Problem and Motivation
The cloud version worked. Groq's Whisper API transcribed Hungarian speech reliably, OpenAI's GPT interpreted the commands, and my lights responded in about four seconds. But four seconds is actually the good news. The bad news is in the variance: the same system took anywhere from 2.7 to 9.2 seconds depending on cloud load and network conditions. On a bad day, it felt slow. On a very bad day — like the one data point at 9.2 seconds — it felt broken.
More fundamentally, I was uncomfortable with what was being sent out. Every voice command I spoke, along with the full list of my smart home devices (names, locations, current states), went to Groq and OpenAI. That is not a privacy disaster, but it is a privacy trade-off I did not need to make.
The other motivation was simply learning. I worked as a mechanical engineering group lead and I am using a career break to build hands-on AI and data science skills. Running local LLMs and STT models myself, understanding where the bottlenecks are, benchmarking performance — this was exactly the kind of project that teaches things you cannot learn from tutorials alone.
System Architecture
The setup spans two machines on a wired home LAN.
The Home Server is a passive-cooled Intel Celeron N3150 box running Debian 12. It has no GPU, runs 24/7, and hosts the orchestration layer: n8n for workflow automation, Domoticz as the smart home controller, and a Mosquitto MQTT broker. Think of it as the brain that coordinates but never does heavy computation.
The Desktop PC is an Intel Core i7-4770 machine running Ubuntu 22.04. This is the AI inference machine. Its GPU changed over the course of the project — first a GTX 1050 Ti with 4 GB VRAM, later an RTX 4060 Ti with 16 GB — and that GPU upgrade is the turning point of the story.
Here is what happens when I press record on my phone:
- I speak a Hungarian command into the Webhook Audio Recorder app, which sends the audio file via HTTP POST to n8n
- n8n sends the audio to faster-whisper for speech-to-text transcription
- In parallel, n8n queries Domoticz for the current device list and their states
- The transcribed text, device list, and current time are passed to Ollama (Qwen2.5:7b), which interprets the command and returns a JSON control payload
- n8n publishes that JSON to the MQTT broker
- Domoticz receives the MQTT message and executes the command — lights go on, blinds move
The AI models I used throughout: Qwen2.5:7b (Q4_K_M quantization, 4.7 GB) for language understanding and JSON generation, and Systran/faster-whisper-small (~500 MB) for Hungarian speech recognition.
Version 1 — Cloud Baseline
The cloud version was straightforward to set up. In n8n, an HTTP Request node calls the Groq Whisper API with the audio file, and an OpenAI Chat Model node handles the LLM side. Domoticz provides the device list, the workflow builds a system prompt, and the AI returns a JSON array of commands.
It worked well. Both the STT and the LLM coped with Hungarian syntax and device names without special tuning — better than I expected. The median end-to-end latency across 21 test runs was 4.0 seconds.
The catch: that 4.0 seconds is the median, not the ceiling. The cloud had a wide spread. OpenAI's response time ranged from 1.6 to 8.2 seconds in my measurements, dragging the total anywhere from 2.7 to 9.2 seconds. Cloud services have their own load and queuing behavior, and my home automation latency was subject to it.
The other catches: cost (paid API subscriptions), internet dependency (no voice control during outages), and the privacy trade-off described above.
Version 2 — Going Local with GTX 1050 Ti
The GTX 1050 Ti has 4 GB of VRAM. That sounds like enough — the Qwen2.5:7b model is 4.7 GB in Q4_K_M quantization. It is not enough.
Ollama loaded approximately 24 of the model's 29 layers into VRAM (~3,500 MiB used). The remaining 5 layers ran on CPU and RAM. This hybrid mode works, but it means every inference cycle crosses the VRAM/RAM boundary repeatedly. The LLM ran at about 3,100 ms per request in warm state — measurable, but acceptable.
The real problem was faster-whisper. After Ollama took 3,500 of the 4,096 MiB available, there was only ~535 MiB of free VRAM left — not enough for the faster-whisper model. I tried the CUDA image anyway and got an immediate "CUDA out of memory" error. There was no other option: faster-whisper ran on CPU.
On this machine, CPU-mode STT took about 2,800–3,500 ms per request. That single constraint — no room in VRAM for the second model — doubled the latency of every request.
The first measurement run with both models running showed a median end-to-end time of 13.3 seconds. Usable, but not satisfying.
The KEEP_ALIVE Discovery
Then I found the single configuration change that cut the response time nearly in half.
By default, Ollama loads the model into VRAM on the first request and unloads it after 5 minutes of inactivity. Every "cold" request — the first one after a quiet period — paid a ~12 second loading penalty. Setting OLLAMA_KEEP_ALIVE=-1 keeps the model permanently resident in VRAM.
With static loading, the median end-to-end latency dropped to 6.9 seconds. Same hardware, same models, one environment variable. The lesson: configuration matters as much as hardware.
The trade-off is that VRAM stays permanently occupied. On the GTX 1050 Ti, that meant zero headroom for any other GPU workload. On a 16 GB card, it would not be a concern.
Version 3 — GPU Upgrade, RTX 4060 Ti
The GTX 1050 Ti taught me that the bottleneck was VRAM, not the CPU. The RTX 4060 Ti has 16 GB. That changes everything.
With 16 GB available, both models fit comfortably on the GPU simultaneously:
The LLM loaded all 29/29 layers into VRAM — confirmed in the Ollama logs:
load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors: CUDA0 model buffer size = 4168.09 MiB
faster-whisper moved from the CPU image to the CUDA image, and VRAM allocation after both models are loaded: Ollama at 4,892 MiB, faster-whisper at 754 MiB, total 5,654 MiB — leaving 10,426 MiB free. The card is barely breaking a sweat.
The results were immediate. GPU-mode STT dropped from ~2,800 ms to 279 ms (static mode, from standalone benchmark) — a 10x speedup. LLM inference dropped from ~3,100 ms to 586 ms (static mode) — a 5x speedup. With static loading enabled, the median end-to-end latency from the n8n measurements was 1.6 seconds.
The cloud baseline was 4.0 seconds. Local AI, on hardware I already owned plus a mid-range GPU upgrade, is now 2.4× faster.
Benchmark Results
All measurements come from real n8n workflow runs — not synthetic benchmarks. The workflow measured the actual time between sending the audio file and receiving the JSON command back, including all network hops between Home Server and Desktop PC.
Full statistics from the raw JSONL data:
| Configuration | n | STT median | LLM median | Total median | Total range |
|---|---|---|---|---|---|
| Cloud (Groq + OpenAI) | 21 | 0.44 s | 2.98 s | 4.0 s | 2.7 – 9.2 s |
| GTX 1050 Ti · dynamic LLM | 17 | 3.54 s | 9.26 s | 13.3 s | 13.2 – 14.3 s |
| GTX 1050 Ti · static LLM | 16 | 2.76 s | 3.48 s | 6.9 s | 6.7 – 7.5 s |
| RTX 4060 Ti · dynamic LLM | 16 | 0.86 s | 2.97 s | 4.4 s | 4.2 – 4.9 s |
| RTX 4060 Ti · static LLM | 58 | 0.34 s | 0.82 s | 1.6 s | 1.5 – 2.1 s |
A few things stand out:
Cloud variance is real. The local GTX configurations had extremely tight variance — the GTX dynamic spread was only 1.1 seconds across 17 measurements. The cloud had a 6.5-second spread. A home automation command that might take 3 seconds or 9 seconds is a different user experience than one that reliably takes 6–7 seconds.
The RTX dynamic mode is interesting. With the RTX 4060 Ti but without static loading, the LLM median was 2.97 seconds — nearly identical to the cloud's 2.98 seconds. The GPU is fast enough that even with model loading overhead amortized across a few requests, you are in the same ballpark as cloud. Enable static loading and you leave cloud performance far behind.
The ~0.5 second overhead is consistent. Across all five configurations, the difference between (STT + LLM) and the total end-to-end time was 0.47–0.63 seconds. That is the n8n workflow overhead plus the local network round-trip. It does not scale with model speed — it is a fixed cost.
Component-Level Numbers
| Metric | GTX 1050 Ti | RTX 4060 Ti | Speedup |
|---|---|---|---|
| STT (faster-whisper-small) | 2,957 ms (CPU) | 279 ms (GPU) | ~10.6× |
| LLM static (Qwen2.5:7b) | 3,079 ms (hybrid) | 586 ms (full GPU) | ~5.3× |
| VRAM used (both models) | ~3,500 MiB / 4,096 total | 5,654 MiB / 16,380 total | — |
Component times from direct benchmark scripts; end-to-end totals from n8n measurement JSONL files.
Key Takeaways
VRAM is the main bottleneck — not the model. The same Qwen2.5:7b model ran in 3,100 ms on GTX (hybrid mode) and 586 ms on RTX (full GPU). The model did not change. The hardware headroom did.
Configuration matters as much as hardware. The single OLLAMA_KEEP_ALIVE=-1 setting cut response time from 13.3 to 6.9 seconds on the GTX — without any hardware change. If you are running Ollama and wondering why it feels slow, check this setting first.
Local AI can beat cloud with the right setup. The RTX 4060 Ti with static loading achieves 1.6 seconds median end-to-end. Cloud median was 4.0 seconds. Local is 2.4× faster — and far more consistent.
Privacy is not a trade-off here. Every voice command, every device state query, every AI inference step stays on the local network. Nothing leaves the house. This is not "good enough for a home project" privacy — it is architecturally private by design.
Open-source models handle minority languages better than expected. Qwen2.5:7b correctly interpreted Hungarian voice commands and in most cases generated valid JSON control payloads across all test configurations. faster-whisper-small transcribed Hungarian speech accurately enough for a smart home context. Neither model was fine-tuned for Hungarian — they work out of the box.
Conclusion
This started as a learning project with modest ambitions: replace cloud APIs with local models, see how the numbers compare, write it up. It ended with a home automation system that responds to Hungarian voice commands in 1.6 seconds, runs entirely offline, and costs nothing per query.
The hardware path matters. A 4 GB GPU creates forced trade-offs; a 16 GB GPU removes them. But the path from 4 GB to 16 GB taught me more about bottlenecks, configuration, and the gap between "it runs" and "it runs well" than any tutorial could.
If you are thinking about building something similar: start with whatever hardware you have. The constraints will teach you something. Then upgrade only what the data tells you to.
If you have questions, suggestions, or a similar build of your own, I would love to hear about it in the comments.
About the Author
Szilárd Galambos spent 20 years as a mechanical engineering group lead at Robert Bosch, and is currently on a deliberate career break to build expertise in data science and AI. With a background in engineering mathematics and hands-on experience in n8n workflow automation, Linux server administration, and AI integration, he bridges the gap between traditional engineering thinking and modern data-driven approaches.
Interests: home automation, AI-powered workflows, and making technology work in the real world.
