September 17, 2026
You do not need a workstation with 128 GB of memory to start using local AI. A modest computer can run smaller language models, while a compatible graphics card can make the experience more responsive.
The difficulty is understanding which memory matters. A computer advertised with 32 GB of RAM and a graphics card with 8 GB of VRAM has two different memory pools. A machine with unified memory follows another design.
To choose sensibly, start with the model, its precision and the amount of information you want it to process. This guide covers inference: running an existing language model, rather than training one. Image and video generation have different requirements.
RAM, VRAM and unified memory serve different roles
System RAM is the working memory used by the operating system, applications and CPU. A model can run from system RAM using CPU inference.
VRAM is the memory attached to a discrete graphics card. When the model and its working data fit there, a compatible inference engine can perform the workload on the GPU.
Unified memory is a shared pool accessible to the CPU and GPU. NVIDIA’s DGX Spark, for example, specifies 128 GB of coherent unified system memory. That is a different arrangement from a desktop with separate system RAM and graphics memory. NVIDIA’s specifications.
The operating system and other applications still need memory. The amount an inference engine can actually allocate also depends on the platform and its software limits.
Consequently, adding 32 GB of RAM to 8 GB of VRAM does not create a single 40 GB graphics-memory pool. Some software can distribute work across both, but that does not make the two arrangements equivalent.
Start by estimating the model’s weights
A language model stores learned numerical values called parameters. Model names often express their approximate number in billions: 8B, for example, means roughly eight billion parameters.
The precision used to store those values strongly affects memory requirements. Quantization reduces the number of bits used for many weights. Local inference engines such as llama.cpp support several quantized formats, including 4-bit and 8-bit options. llama.cpp documentation.
A useful first calculation is:
Raw weight storage in GB ≈ billions of parameters × bits per parameter ÷ 8
The following figures are our arithmetic estimates, assuming every parameter uses exactly the stated precision.
| Total parameters | 16-bit weights | 8-bit weights | 4-bit weights |
|---|---|---|---|
| 3 billion | 6 GB | 3 GB | 1.5 GB |
| 8 billion | 16 GB | 8 GB | 4 GB |
| 14 billion | 28 GB | 14 GB | 7 GB |
| 32 billion | 64 GB | 32 GB | 16 GB |
| 70 billion | 140 GB | 70 GB | 35 GB |
These are weight-only estimates, not complete hardware requirements. They use decimal GB; some monitoring tools report binary GiB instead.
Real quantized files also contain supporting information, and some tensors may retain higher precision. Runtime buffers and conversation data require additional space. A 32B model’s theoretical 16 GB of 4-bit weights therefore should not be read as a promise that it will run entirely on a 16 GB GPU.
Quantization also introduces a quality trade-off. Test the exact model variant on your work: a smaller file is useful only if its answers remain good enough.
The conversation needs memory too
For many transformer models, a KV cache stores intermediate information from previously processed tokens. This avoids repeating some calculations during generation, but consumes memory.
Longer conversations and larger documents can increase this requirement. The exact behaviour depends on the architecture and cache strategy; sliding-window layers, for example, can stop growing once their window is full. Hugging Face documents options including cache offloading and quantization, each with performance trade-offs. Hugging Face’s cache guide.
This explains why a model may handle a short question successfully and struggle with a large document. The weights are unchanged, but the working data is different.
For planning purposes, budget for:
Weights + context cache + runtime buffers + space for the rest of the system.
A model’s advertised maximum context length describes a supported capability. It does not guarantee that your computer has the memory to use that maximum.
Practical starting points for GPU memory
The table below gives editorial planning estimates, not measured benchmarks or universal compatibility guarantees. It assumes a compatible engine, one text-generation session, approximately 4-bit weights and a modest context of roughly 4,000 tokens. Architecture and quantization format can change the outcome.
| Dedicated VRAM | Model sizes worth investigating first | Main caution |
|---|---|---|
| 6–8 GB | Smaller models around 3B–8B | The upper end leaves less room for context and other GPU use |
| 12–16 GB | Models around 8B–14B | Check the exact quantized variant and runtime allocation |
| 24 GB | Models around 20B–32B | Long contexts may still exceed available memory |
| 48 GB | Some 70B-class quantized models | Fit depends on cache size, format and reserved memory |
These ranges follow from the weight estimates above with additional headroom. They are a shortlist for testing, rather than a reason to buy hardware without checking a specific model.
For example, compare an 8B model with 4 GB of theoretical 4-bit weights against a 14B model with 7 GB. On an 8 GB GPU, the first leaves substantially more room for everything beyond the raw weights. Neither figure alone establishes the final memory footprint.
How much system RAM should you plan for?
For a general-purpose computer, our starting recommendation is 16 GB for experimenting with small models, or 32 GB for more room alongside everyday applications. These are practical budgeting suggestions, not minimum requirements imposed by an AI runtime.
CPU-based inference and partial GPU offloading can require considerably more RAM when larger models are involved. Size that RAM against the actual allocation, with room for your browser, editor and operating system.
llama.cpp explicitly supports hybrid CPU/GPU inference for models larger than available VRAM. This can make a larger model usable, but speed will depend on the split and hardware. llama.cpp’s supported capabilities.
Try a representative task before upgrading. Being able to load a model and being comfortable waiting for its answers are separate tests.
Do 128 GB or 192 GB make sense?
Large shared-memory configurations become interesting when you have identified a model or workload that exceeds smaller machines’ capacity. They may provide room for larger weights, more context or several simultaneous workloads.
Consider a hypothetical 120B model: at exactly 4 bits per parameter, its weights alone represent 60 GB. That calculation helps explain why a larger memory pool can matter. It does not establish the requirements of any particular 120B model.
Moving from 128 GB to 192 GB adds capacity, but capacity alone does not predict generation speed. NVIDIA lists memory capacity and bandwidth separately for DGX Spark: 128 GB and 273 GB/s. Those specifications describe different properties. NVIDIA’s specification table.
Our recommendation is to justify that extra capacity with a named model, a chosen format and a realistic workload. For short conversations with a small model, it may remain largely unused.
Watch out for “active parameters” in MoE models
Mixture-of-experts models can activate only part of their parameters for each token. The smaller active count must not be substituted for the full model size when estimating resident weights.
For example, Qwen lists 30.5 billion total parameters and 3.3 billion activated parameters for Qwen3-30B-A3B. Official Qwen model card.
At an idealized 4 bits per parameter, those totals correspond to roughly 15.25 GB of weights, not 1.65 GB. This is a calculation, not a measured deployment footprint. Specialized offloading can change where weights reside, but the active count alone is not a memory requirement.
For a larger-scale example, our Tencent Hy4 Preview explainer examines the distinction between total and active parameters, and why it matters for deployment.
Check your own workload before buying
With Ollama, run ollama ps while a local model is loaded. Its processor column indicates CPU, GPU or mixed placement. Ollama also documents additional memory requirements for parallel requests and multiple loaded models. Ollama FAQ.
Use a repeatable test:
- Choose the exact model and quantized version you intend to use.
- Start with one session and a moderate context setting.
- Record memory use, time to the first answer and overall completion time.
- Repeat with a document or coding task representative of your daily work.
- Increase context or concurrency only when the use case requires it.
If a model struggles, compare a smaller variant before assuming that more hardware is the only solution. Also confirm that execution is genuinely local: Ollama supports cloud-hosted models too, which do not provide a test of your computer’s inference capacity.
If you are weighing a hardware upgrade against remote execution, our guide to Ollama Cloud pricing and usage credits explains how to estimate the cost of that alternative.
The useful target is enough memory for the model that handles your work well, with space for its working data and the rest of your applications. A configuration that produces dependable answers at an acceptable speed is a stronger choice than one selected solely for the largest model it can load.
Sources
- llama.cpp — supported hardware, quantization and hybrid CPU/GPU inference
- Hugging Face Transformers — cache strategies and memory trade-offs
- Ollama — memory placement, concurrent requests and local/cloud execution
- NVIDIA — DGX Spark unified memory and technical specifications
- Qwen — Qwen3-30B-A3B model card