Why Run AI Locally?
Privacy, cost, and control. No API keys, no rate limits, no data leaving your machine. For developers, researchers, and privacy-conscious users, local AI is increasingly viable — and 2026 is the year it went mainstream.
Hardware Requirements
The golden rule: model size in GB ≈ VRAM needed for decent performance. Here's the real-world breakdown:
Minimum Viable Setup
- CPU-only: Any modern CPU with 16GB+ RAM can run 7B models at 2-5 tokens/second. Slow but functional.
- GTX 1660 (6GB): 3B-4B models at 20-30 tok/s. Good enough for chatbots and simple tasks.
- RTX 3070 (8GB): 7B-8B models at 30-50 tok/s. The sweet spot for most users.
- RTX 4090 (24GB): 13B-34B models at 40-60 tok/s. Near-API quality.
- Apple M2/M3 with 32GB+: 7B-13B models at 15-25 tok/s using MLX. Excellent efficiency.
An RTX 3070 (8GB) running Qwen 2.5 7B Q4_K_M uses ~5.2GB VRAM and generates 45 tokens/second — comparable to GPT-3.5 speeds for many tasks.
The Tools
Ollama
The easiest way to get started. ollama run llama3.1 and you're chatting. Supports GGUF models, GPU acceleration, and a REST API. The ecosystem of 5,000+ community models makes it the de facto standard for local LLMs in 2026.
llama.cpp
The engine underneath most local AI tools. Written in C++, supports CPU, CUDA, Metal, Vulkan, and ROCm. If you need maximum performance or want to embed LLMs in an application, this is the foundation.
LM Studio
GUI-focused, excellent for beginners. Point-and-click model downloading, chat interface, and API server. Good for exploring before committing to CLI tools.
vLLM
For serving models at production scale. OpenAI-compatible API, continuous batching, PagedAttention for efficient memory usage. Overkill for personal use but what you'd use to build a service.
Model Recommendations (August 2026)
- Best all-rounder (8GB VRAM): Qwen 3.5 7B — strong reasoning, good coding, permissive license
- Best coding (8GB VRAM): DeepSeek Coder V2 7B — specialized for code generation and analysis
- Best small model (4GB VRAM): Llama 3.2 3B — surprisingly capable for its size
- Best for Mac (Apple Silicon): MLX-optimized models like Qwen 3.5 MLX — leverage the unified memory architecture
- Best for roleplay/creative: Mistral Nemo 12B — strong narrative capabilities
Quantization Explained
Models are trained in 16-bit precision (FP16). A 7B parameter FP16 model takes 14GB. Quantization compresses these weights:
- Q8_0: 8-bit. Near-zero quality loss. ~7GB for 7B model.
- Q4_K_M: 4-bit with mixed precision. 3-5% quality loss. ~4.5GB for 7B model. The standard choice.
- Q2_K: 2-bit. Noticeable quality loss. Only for extreme memory constraints.
Real Performance Numbers
Testing on an RTX 3070 (8GB) with Qwen 3.5 7B Q4_K_M:
- Basic chat: 48 tok/s
- Code generation: 42 tok/s
- Summarization (2K context): 38 tok/s
- Long context (32K): 25 tok/s
For most use cases, this is indistinguishable from cloud API latency — and it's free, private, and always available.