# Converting parameters to VRAM: calculating memory

[Skip to content](#lm-inhoud)Network/[NL](/en/parameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fhub.llmnet.nl%2Fen%2Fparameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig&text=Converting%20parameters%20to%20VRAM%3A%20calculating%20memory)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fhub.llmnet.nl%2Fen%2Fparameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig)[](https://www.reddit.com/submit?url=https%3A%2F%2Fhub.llmnet.nl%2Fen%2Fparameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig&title=Converting%20parameters%20to%20VRAM%3A%20calculating%20memory)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fhub.llmnet.nl%2Fen%2Fparameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig&text=Converting%20parameters%20to%20VRAM%3A%20calculating%20memory)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fhub.llmnet.nl%2Fen%2Fparameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig)[](https://www.reddit.com/submit?url=https%3A%2F%2Fhub.llmnet.nl%2Fen%2Fparameters-omrekenen-naar-vram-hoeveel-geheugen-heb-je-nodig&title=Converting%20parameters%20to%20VRAM%3A%20calculating%20memory)[](#)

 
# Converting parameters to VRAM: how much memory do you need

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 Selecting hardware for open-source language models almost always begins with the same question: does the chosen model fit within the available video memory (VRAM)? Those who only look at the parameter count (in billions) and multiply it by the number of bytes per value often find themselves unpleasantly surprised in practice. A 70B model might theoretically fit into 35 gigabytes at 4-bit precision, but will immediately crash with an Out-Of-Memory (OOM) error as soon as a user submits a long document.

 The total memory footprint of an LLM during inference consists of four distinct components: the static model weights, the dynamic Key-Value cache (KV cache), the intermediate activation vectors, and the runtime framework overhead. To understand why long input texts place an exponential demand on memory, it is advisable to study [how a context window works](https://hub.llmnet.nl/en/context-window-uitleg) where the relationship between attention mechanisms and token processing is explored in greater depth. In this article, the complete mathematical calculation method is dissected so you can predict the required VRAM down to the megabyte.

 
## The basic formula for model weights

 The most prominent consumer of video memory is the set of trained weights of the neural network. Each weight in the network is stored in a specific numeric format. The amount of memory required purely to load the static weights can be calculated using a simple linear equation:

 Geheugen_gewichten (in GB) = (Aantal_parameters in miljarden * Bits_per_gewicht) / 8

 In an uncompressed state, modern models are typically trained in 16-bit floating point (FP16 or BF16). Because 16 bits equal 2 bytes, an 8-billion parameter (8B) model in FP16 requires exactly 16 gigabytes of storage just to sit in memory. If the same model is loaded in 8-bit precision (1 byte per parameter), this footprint is halved to 8 GB. With popular 4-bit quantization (0.5 bytes per parameter), the weight drops to 4 GB.

 In reality, this number is always about five to ten percent higher due to the presence of non-quantized layers (such as the input embeddings and the final output normalization heads), quantization scales, and metadata. As a result, an 8B parameter model file in 4-bit format occupies roughly 4.5 to 4.9 GB on disk and in memory, rather than the theoretical 4.0 GB.

 
## Quantization in practice: GGUF, AWQ, GPTQ, and EXL2

 Because consumer graphics cards and SME workstations often feature 8, 16, or 24 GB of VRAM, quantization has become the standard procedure for running medium and large models locally. Different quantization methods each strike their own balance between compression ratio, inference speed, and preservation of logical reasoning capabilities.

 For CPU and hybrid inference via tools such as llama.cpp, the GGUF format is dominant. This approach utilizes so-called 'k-quants' (such as Q4_K_M or Q5_K_S), where critical attention matrices are preserved at higher precision (e.g., 5 or 6 bits) while less sensitive feed-forward layers are reduced to 4 bits. Anyone considering deploying compact models directly on local hardware or workstations can consult the guide on [on-device implementation of small models](https://hub.llmnet.nl/en/kleine-modellen-op-apparaat) to read how these k-quants perform on consumer hardware.

 On dedicated Nvidia GPUs, methods such as AWQ (Activation-aware Weight Quantization) and EXL2 (ExLlamaV2) deliver significantly higher processing speeds per second. AWQ protects the top one percent of most important weights from precision loss, enabling 4-bit inference to achieve virtually identical benchmark scores to the original 16-bit base model. EXL2 even allows for fractional bits (such as 3.5 or 4.25 bits per weight), making it possible to tailor a model precisely to a specific VRAM capacity, such as 24 GB.

 
## The forgotten factor: calculating the KV cache

 Once the model weights are loaded into memory, the model is not yet ready to answer prompts. During text generation, the attention mechanism (Self-Attention) computes a Key and Value vector for every token in the input and output. To avoid recomputing the entire document from start to finish with each newly generated word, these vectors are stored in the Key-Value cache (KV cache).

 The KV cache scales strictly linearly with context length and the number of concurrent users (batch size). The exact formula for the KV cache memory footprint of a traditional Multi-Head Attention (MHA) model is:

 KV_cache_grootte (bytes) = 2 * Lagen * Heads * Dimensie_per_head * Contextlengte * Batch_size * Bytes_per_element

 Here, the factor 2 represents the two distinct vectors (Key and Value). Modern architectures almost universally employ Grouped-Query Attention (GQA) or Multi-Query Attention (MQA). With GQA, multiple query heads share the same KV head, reducing the memory footprint of the cache by a factor of 4 to 8. As a concrete calculation example, let us take a model with 32 layers, 8 KV heads (via GQA), and a head dimension of 128, running on FP16 (2 bytes) with a context of 32,768 tokens at batch size 1:

 KV-cache = 2 * 32 * 8 * 128 * 32768 * 1 * 2 bytes
 = 4.294.967.296 bytes = 4,00 GB

 This calculation directly highlights the danger: loading an 8B model in 4-bit (4.5 GB weights) and opening a context window of 32k tokens requires 4.0 GB of additional memory for the KV cache alone. If the context is expanded to 128k tokens, the KV cache explodes to 16 GB, causing dynamic memory usage to far exceed the static model weights.

 
## Activations, context overhead, and runtime reserves

 In addition to weights and the KV cache, the inference engine claims memory for activations: the temporary tensors created during matrix multiplications across each transformer layer. In pure autoregressive inference with a batch size of 1, this consumption is relatively modest (between 200 and 800 MB), but during prefilling (processing a massive 50,000-token prompt all at once), activation memory can suddenly spike by several gigabytes.

 Furthermore, the software stack itself claims a non-negligible amount of VRAM:
 - CUDA context: Nvidia's runtime environment reserves between 300 MB and 800 MB of VRAM by default as soon as PyTorch or vLLM initializes.
 - Memory fragmentation: Paging and dynamic allocations cause gaps in memory. Without optimized memory managers (such as PagedAttention in vLLM), 10 to 20 percent of effective capacity is lost as a result.
 - Compute buffers: Kernels for specific operations (such as FlashAttention-3) allocate scratchpads for matrix computations.

 As a rule of thumb, a safety headroom of at least 1.5 to 2.0 GB should always be added to the sum of the weights and the KV cache to prevent unexpected OOM crashes during peak loads.

 
 
 
 
 Model Size | 
 Precision | 
 Weights (VRAM) | 
 KV Cache (8k tokens, GQA) | 
 KV Cache (32k tokens, GQA) | 
 Minimum VRAM (32k context) | 
 

 
 
 
 7B / 8B | 
 4-bit (Q4/AWQ) | 
 ~4.8 GB | 
 ~1.0 GB | 
 ~4.0 GB | 
 10.5 GB | 
 

 
 7B / 8B | 
 8-bit (FP8/INT8) | 
 ~8.6 GB | 
 ~1.0 GB | 
 ~4.0 GB | 
 14.5 GB | 
 

 
 14B | 
 4-bit (Q4/AWQ) | 
 ~8.9 GB | 
 ~1.5 GB | 
 ~6.0 GB | 
 16.5 GB | 
 

 
 32B | 
 4-bit (Q4/AWQ) | 
 ~19.2 GB | 
 ~2.0 GB | 
 ~8.0 GB | 
 29.5 GB | 
 

 
 70B | 
 4-bit (Q4/AWQ) | 
 ~39.5 GB | 
 ~2.5 GB | 
 ~10.0 GB | 
 52.0 GB | 
 

 
 
 

 
## Real-world scenarios calculated: from 8B to 70B

 To translate the theory into tangible hardware choices, we walk through three common architectural scenarios.

 
### Scenario A: 8B model on a single 16 GB GPU (Nvidia RTX 4080 / T4)

 A developer wants to run a Llama-3-8B model for document analysis. In 4-bit AWQ, the weights take up 4.8 GB. With a target context length of 16,000 tokens, the GQA KV cache requires 2.0 GB. CUDA overhead and framework reserves amount to 1.5 GB. The total memory demand is 4.8 + 2.0 + 1.5 = 8.3 GB VRAM. This fits comfortably on a 16 GB graphics card. There is even enough headroom to increase the batch size to 4 concurrent requests (the KV cache then scales to 8.0 GB, with total usage at 14.3 GB).

 
### Scenario B: 32B model on a single 24 GB GPU (Nvidia RTX 3090 / 4090)

 A Qwen-2.5-32B model in 4-bit precision requires approximately 19.2 GB for the weights. Adding 1.5 GB of base overhead leaves only 3.3 GB of free space for the KV cache. However, at a full 32k context, the KV cache demands 8.0 GB. Result: the model will inevitably crash with a context larger than roughly 8,000 tokens. The solution here is either to switch to more aggressive 3-bit EXL2 quantization (reducing weights to ~15 GB) or to enable FP8 KV cache quantization to halve the cache memory footprint.

 
### Scenario C: 70B model on 2x 24 GB GPUs (48 GB total)

 A Llama-3-70B model in 4-bit quantization requires 39.5 GB for weights. Distributed across two cards via Tensor Parallelism, that is approximately 20 GB of weights per card. With a combined 48 GB, roughly 5.5 GB remains for the total KV cache and activations. This is sufficient for a context up to around 12,000 tokens at batch size 1. For intensive production workloads with 32k or 64k context, a minimum of two 32 GB or 48 GB cards (such as the Nvidia A6000 or L40S) is required.

 
## Multi-GPU setups: Tensor Parallelism versus Pipeline Parallelism

 When a model does not fit onto a single graphics card, the workload must be distributed across multiple processors. The two primary strategies for this have a direct impact on memory distribution:

 With Tensor Parallelism (TP) individual weight matrices are sliced horizontally or vertically across multiple GPUs. Each GPU executes a portion of the computation concurrently. As a result, not only are model weights distributed perfectly evenly, but the number of attention heads is also split, reducing the KV cache footprint per GPU. The drawback is that TP demands extremely high-speed interconnects between the cards (preferably NVLink); over standard PCIe buses, synchronization overhead can significantly degrade processing throughput.

 With Pipeline Parallelism (PP) consecutive transformer layers are assigned to consecutive cards (for example, layers 1-40 on GPU 0 and layers 41-80 on GPU 1). While this requires less bandwidth between cards, it introduces so-called pipeline bubbles (idle wait times), and each GPU must maintain sufficient reserve memory for passing activations forward.

 
## Local hardware or migrating to hosted infrastructure

 The math behind VRAM makes it clear that hardware requirements scale exponentially as models exceed 32 billion parameters or when long contexts of 100k+ tokens are standard requirements. For many organizations, procuring and maintaining dedicated GPU clusters represents a substantial cost. Those looking to weigh the total cost of on-premise hardware against scalable cloud solutions can consult the in-depth cost overview in the article on [the total cost of ownership (TCO) of open vs. closed models](https://hub.llmnet.nl/en/tco-open-vs-closed) for concrete infrastructure comparisons.

 When strict confidentiality, compliance, and data sovereignty are paramount, running locally often remains the only permitted route. The overview of [AI models and privacy under the GDPR](https://hub.llmnet.nl/en/ai-modellen-en-privacy-avg-compliance) explains in detail under which legal conditions local processing is mandatory to prevent data leaks to external cloud services.

 If flexibility is the primary priority and workloads fluctuate significantly throughout the day, combining multiple specialized API providers via a routing layer can be considerably more cost-effective than maintaining over-provisioned local hardware. To understand how automated failover and cost optimization across dozens of model providers are structured, the guide on [how an LLM API aggregator works](https://api.llmnet.nl/en/aggregator-uitleg) provides insight into advanced gateway architectures.

 
## VRAM dynamics in audio and multimodal models

 Not all neural networks exhibit the exact same memory characteristics as pure autoregressive text models. Multimodal models (Vision-Language Models) and audio networks employ different attention mechanisms and encoders that impose unique demands on video memory.

 With vision models (such as Qwen2-VL or Llama-3.2-Vision), an image is split into patches that are converted into hundreds or thousands of image tokens. A single high-resolution image can therefore instantly add 2,048 tokens to the context. This triggers a sharp spike in KV cache memory. Additionally, the Vision Encoder (often a ViT architecture) must remain resident in VRAM simultaneously, which typically adds 1.0 to 2.5 GB of permanent weight overhead.

 In audio transcription and speech generation, models frequently use an Encoder-Decoder structure (such as Whisper) or diffusion-based architectures. Those wanting to explore how these architectures differ from classical language models in terms of memory pressure and latency can find in-depth technical specifications in the article on [specialized audio and music models](https://hub.llmnet.nl/en/audio-en-muziek-modellen), which details the distinct memory requirements of spectrogram processing.

 
## Profiling methods and techniques to reduce memory consumption

 Those looking to optimize their memory budget without compromising on model size have access to advanced optimization techniques in modern runtime engines such as vLLM, TensorRT-LLM, and SGLang:

 1. KV Cache Quantization (FP8 and INT4): By storing Key and Value vectors in 8-bit float (FP8 E4M3 or E5M2) or 4-bit integer instead of 16-bit float, context memory usage is cut in half or reduced by a factor of four. The quality loss with FP8 KV cache is negligible on standard benchmarks (<0.5%), while immediately freeing up gigabytes of VRAM for longer prompts or larger batch sizes.

 2. PagedAttention and Chunked Prefill: PagedAttention divides the KV cache into virtual pages (similar to paging in operating systems). This completely eliminates internal memory fragmentation, boosting VRAM utilization from around 70% to more than 96%. Chunked prefill splits massive input prompts into manageable chunks, smoothing out activation spikes during prompt processing.

 3. Measuring accurately in production: During testing, never rely solely on simple tools like nvidia-smi, because frameworks like PyTorch pre-allocate large blocks of memory via their own caching allocator. Instead, use the inference engine's internal profiling tools (such as torch.cuda.memory_allocated() versus torch.cuda.memory_reserved()) to distinguish actual data usage from reserved buffers.

 By systematically adding up the static weights, the quantized KV cache, the peak activation level, and a fixed headroom of two gigabytes, any deployment pipeline can be accurately dimensioned in advance. This prevents costly GPU investments from falling short or being unnecessarily oversized.