The idea of running a local LLM (Large Language Model) has always appealed to me, especially concerning data privacy and cost control. However, when I first delved into this, I realized through my own experiences how misleading market claims like "a few GB of RAM is enough" can be. In real-world scenarios, running a 70B parameter model with 8GB of VRAM is only possible with significant optimizations, which come with certain trade-offs.
In this post, I will share my experiences, the problems I encountered, and the solutions I found, from hardware selection to optimization techniques for local LLMs. My goal is to offer a concrete, practical, and "good enough" perspective to anyone interested in this field. As we begin, we must remember that VRAM is the most critical part of this equation.
VRAM: The Heart of Local LLMs and Capacity Limits
At the core of running an LLM locally is keeping the model's weights in the GPU's VRAM. As the model size grows, the amount of VRAM it needs naturally increases. For example, a 7 billion parameter (7B) model in 16-bit float (FP16) format requires about 14GB of VRAM, while a 70B parameter model can demand up to 140GB. These values are far beyond the hardware owned by an average user.
While working on AI-powered operations for my side product and a production planning model for a client project, I had the opportunity to experiment with models of different sizes. I clearly saw that there can sometimes be differences between theoretical VRAM requirements on paper and practical usage, especially as the context window grows. A 7B model, with a common quantization like Q4_K_M, can generally run with around 5-6GB of VRAM. However, for a 13B model, this value jumps to 8-10GB, and for a 70B model, it can soar to 40-50GB. This also varies depending on parameters like context window and batch size.
💡 VRAM Monitoring Tips
You can monitor the real-time status of your GPU and VRAM with the
nvidia-smicommand. Usingwatch -n 1 nvidia-smito update VRAM usage every second will help you understand how much memory is consumed when loading a model or performing inference.
While 8GB or 12GB VRAM cards are common in the market, running large models like 70B on these cards requires more than just VRAM; significant optimizations like quantization are essential. Sometimes, even running a 7B model with full performance and long contexts can be challenging with 8GB of VRAM. At this point, not only the VRAM capacity but also the memory bandwidth of the GPU becomes important. Higher bandwidth allows model weights to be read and processed faster, increasing inference speed. If fitting a model into VRAM is an achievement, making it run fast afterward is another challenge.
Quantization: Gaining Speed from Memory and Quality Trade-offs
Quantization is a lifesaver for those of us who want to run LLMs locally. Essentially, it means representing the model's weights using fewer bits. For example, using int8 (8-bit) or int4 (4-bit) instead of float16 (16-bit) significantly reduces model size and thus VRAM requirements. This way, I can run a 70B model, which would normally require 140GB of VRAM, with 4-bit quantization using around 40GB of VRAM.
In my experience, especially with GGUF format models used by projects like llama.cpp, quantization levels like Q4_K_M generally offer a good balance. These formats keep the model's performance and output quality at acceptable levels while significantly reducing VRAM consumption. While prompt engineering for a client project, I closely observed the differences in output quality between different quantization levels of the same model. Less compressed formats like Q8_0 yielded better output, but Q4_K_M offered a more practical solution in terms of both performance and memory.
# Example of running a 4-bit quantized model with ollama
# This model comes pre-quantized.
ollama run mistral:7b-instruct-v0.2-q4_K_M
# Example of manually running a 4-bit quantized model with llama.cpp
# You need to have downloaded the model first (e.g., in GGUF format from Hugging Face)
./main -m models/mistral-7b-instruct-v0.2.Q4_K_M.gguf -p "Write me a poem about artificial intelligence." -n 512
Of course, quantization comes with a cost: a potential drop in output quality. Especially in very sensitive or creative tasks, lower bit depths can sometimes lead to meaningless or erroneous outputs. Therefore, choosing which quantization level to use is a trade-off depending on your project's sensitivity and available hardware. I generally prefer the most compressed format that provides the lowest acceptable quality, because often speed and memory savings are more critical than a slight drop in quality. This is part of the "good enough" philosophy; instead of always aiming for perfection, it's about finding the most efficient solution that gets the job done.
Speed Factors: CPU, Storage, and Inference Engines
Limiting local LLM performance to just GPU and VRAM would be a big mistake. Factors like model loading, CPU processing tokens, and efficient inference engine operation play critical roles in overall performance. Especially with large models, disk speed directly affects how quickly model files are loaded into VRAM. There's a world of difference between loading a 40-50GB 70B model file from an HDD versus a fast NVMe SSD. In my tests, NVMe drives can reduce model loading times by up to 70%.
The CPU takes on a significant workload, especially in hybrid CPU/GPU inference engines like llama.cpp. If part of the model doesn't fit into VRAM or if CPU offloading is used, the CPU's core count and speed directly impact inference speed. While integrating LLMs into the backend of my anonymous Turkish data platform, I realized how crucial it was to set the correct thread count for llama.cpp. Excessive thread usage can degrade performance due to context switching costs, while insufficient thread usage wastes CPU resources.
# Setting thread count in llama.cpp (example)
# -t N: Use N CPU threads
./main -m models/mistral-7b-instruct-v0.2.Q4_K_M.gguf -p "Make me a 5-item list." -n 128 -t 8
Inference engines themselves offer an additional layer of optimization. llama.cpp is a popular choice that can efficiently use both CPU and GPU, with broad model support. vLLM, on the other hand, is designed more for high-performance GPUs, increasing throughput with techniques like batching and continuous batching. Which engine you choose depends on your hardware and use case. I generally prefer llama.cpp for its simplicity and flexibility, especially in hybrid systems. By setting CPU and memory limits for a service with cgroup, I ensure that LLM inference doesn't affect other critical services. Last month, when I accidentally caused a service to be OOM-killed by writing sleep 360, I once again understood the importance of cgroup limits.
Hardware Selection and Budget Planning: The "Good Enough" Approach
Hardware selection for local LLMs is directly proportional to your budget and the size of the model you want to run. You don't always have to buy the most expensive card; the important thing is to find the most efficient solution that meets your needs. My approach to this has always been "good enough"; that is, getting the best performance with the available resources.
Here are my observations and recommendations for different budget levels:
- Entry-Level (8-12GB VRAM): If your budget is limited or you just want to experiment with small models (around 7B), an RTX 3060 (12GB VRAM), RTX 4060 (8GB VRAM), or even older generation cards bought second-hand might suffice. With these cards, you can run 7B models smoothly in compressed formats like Q4_K_M. At this level, running 13B models might require much more aggressive quantization or CPU offloading, which would severely reduce speed.
- Mid-Range (16-24GB VRAM): This segment can be the "sweet spot" for many. With cards like the RTX 3090 (24GB VRAM), RTX 4080 Super (16GB VRAM), or RTX 4090 (24GB VRAM), you can comfortably run 13B models and even try some 30B models with Q4_K_M. I've found that the price/performance ratio of second-hand cards like the RTX 3090 can be quite attractive when building my own systems. Especially in the post-crypto mining market, such opportunities can arise.
- High-End (48GB+ VRAM): If you want to run 70B and larger models locally, you either need to opt for professional-grade cards (like NVIDIA A6000 with 48GB VRAM) or combine multiple RTX 4090s to pool VRAM (e.g., with
llama.cpp's multi-GPU support). At this level, costs significantly increase. While working on a larger and more complex LLM model for production planning in an manufacturing company's ERP, we had to conduct a very detailed return-on-investment analysis for such hardware.
ℹ️ Evaluating the Second-Hand Market
In my opinion, exploring the second-hand market, especially for mid-range and high-end cards, can be smart. Prices can be much more affordable than new cards, and with the right choice, you can significantly save on your budget. However, always check the seller's history and, if possible, have the opportunity to test the card. I faced a similar budget and hardware selection dilemma when optimizing
PostgreSQLperformance on a VPS; the most expensive solution is not always the best, the important thing is to find what suits the need.
Remember, LLMs are evolving rapidly, and new optimization techniques are constantly emerging. Therefore, instead of making a huge investment initially, choosing what suits your needs and upgrading over time might be a more sensible strategy.
Practical Application and Optimization Tips
Efficiently running local LLMs on your hardware not only requires selecting the right hardware but also using the right tools and fine-tuning. The two main tools I've used and found most beneficial in this process are ollama and llama.cpp.
ollama offers an incredibly easy interface for running local LLMs. With a single command, you can download and run popular models, and even import your own model. Its API also allows for easy integration into your other applications.
# After installing ollama
# Download and run a model
ollama run llama2:7b
# Start a chat with a different model
ollama run mistral:latest
# Example of sending a request with curl using ollama's API
curl http://localhost:11434/api/generate -d '{
"model": "mistral:latest",
"prompt": "Why are local LLMs important?",
"stream": false
}'
llama.cpp, on the other hand, offers lower-level control and more optimization options. By compiling its source code, you can make hardware-specific optimizations and run different GGUF models directly. Using make -j to compile, utilizing all your CPU cores, significantly shortens compilation time.
# Clone and compile the llama.cpp repository
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j # Automatically adjusts based on your CPU core count
# Example of running a model with the compiled main binary
# -m: model path
# -p: prompt
# -n: number of tokens to generate
# -t: number of CPU threads to use
# --gpu-layers: number of layers to run on GPU (adjusted based on VRAM)
./main -m models/llama-2-7b-chat.Q4_K_M.gguf -p "What are the advantages of using local LLMs?" -n 256 --gpu-layers 30
Performance monitoring and resource management are another point not to be overlooked. Following system logs with journald and limiting resource consumption of LLM services with cgroup are critical for overall system stability. For example, if an LLM service unexpectedly consumes too much memory and gets OOM-killed, you can see this in journald logs and adjust your cgroup settings accordingly. Similarly, using auditd to monitor specific file accesses or system calls can be useful for identifying security and performance issues, especially when an LLM's access to the file system is concerned.
⚠️ Caution with Resource Limiting
Care must be taken when setting resource limits with
cgroup. Too low limits can cause LLM inference to slow down or fail entirely. Finding the right limits requires trial and error and closely monitoringjournaldoutputs.
Playing with parameters like batch size and context window in engines like llama.cpp also affects performance. A larger batch size can increase throughput but also increases VRAM consumption. The longer the context window, the more past information the model can remember, but this also extends inference time. Adjusting these parameters according to your project's needs and your hardware's capacity is a practical reflection of the "good enough" philosophy.
Conclusion
Stepping into the world of local LLMs brings with it some challenges, especially on the hardware side. However, from my own experiences, I've seen that with the right knowledge and approach, it's possible to efficiently run models like 7B on an 8GB VRAM system, and even push to 70B in some cases. Throughout this process, I personally experienced the critical role of VRAM, the saving effect of quantization, and the importance of other factors like CPU and disk speed.
Remember, you don't always need the latest or most expensive hardware. The important thing is to use your available resources in the best way possible to create a solution that is suitable for your project's needs and cost-effective. With a "good enough" approach, you can get maximum efficiency from your current hardware and make smart upgrades when necessary.
