An AI workload can run out of host RAM, GPU VRAM, container memory or application heap. Those failures look similar from the client—a reset, timeout or killed process—but require different fixes.

Do not start by increasing every limit. First identify which memory pool failed and why. For host-level context, see Linux for AI developers and the AI operations hub.

Identify the failed layer

Symptom	Likely layer
Kernel logs show an OOM kill	Host or container RAM
CUDA “out of memory”	GPU VRAM
Exit code 137	Container/cgroup kill, often memory
Node heap allocation failure	JavaScript heap
Kubernetes OOMKilled	Pod exceeded its memory limit
Model fails during load	Weights/runtime exceed RAM or VRAM
Failure grows with context length	KV cache or request buffers

Check evidence close to the failure:

free -h
dmesg -T | grep -i -E 'out of memory|killed process'
nvidia-smi
docker stats
journalctl -u ai-inference --since "20 minutes ago"

Estimate model memory

Model parameter count is only the beginning. Runtime memory includes:

• model weights at their stored precision;
• quantization metadata and runtime buffers;
• KV cache, which grows with context and concurrency;
• framework and CUDA overhead;
• embeddings, rerankers or vision encoders loaded beside the main model;
• request and response buffers.

A model that fits for one short prompt may fail with four concurrent long-context requests. Test representative context lengths and concurrency rather than a single warm-up call.

Reduce VRAM pressure

Use the least destructive change that meets the workload:

1. Lower concurrent sequences or batch size.
2. Reduce maximum context where the product does not need it.
3. Choose a smaller or more aggressively quantized model.
4. Offload selected layers to system RAM if latency remains acceptable.
5. Stop other GPU processes.
6. Move to a larger GPU or split the model across supported devices.

Quantization trades memory for some combination of quality, compatibility and speed. Validate the actual tasks rather than assuming a quantized model is equivalent.

Host RAM and model loading

Large model downloads, conversion and startup can temporarily need more memory than steady-state inference. Avoid loading multiple copies during a rolling deployment unless capacity accounts for both.

Swap may keep a process alive but can make inference unusably slow. It is a safety buffer, not replacement capacity for active weights.

For ingestion pipelines, stream documents and process bounded batches instead of loading an entire corpus into memory. Persist progress so a failure does not restart the whole job.

Containers

Inspect both runtime usage and configured limits:

docker inspect inference --format '{{json .HostConfig.Memory}}'
docker stats inference

A host with free memory does not help a container capped below its workload. Conversely, removing the limit can allow one model to destabilise every service on the host.

Set realistic reservations and limits, expose readiness only after the model is loaded, and stop routing new work before terminating an old instance.

Kubernetes

For an OOMKilled pod, compare actual peak use with requests and limits. Do not simply raise the limit above node capacity.

GPU scheduling and RAM scheduling are separate. A pod can receive a GPU and still lack host memory for tokenization, model loading or preprocessing. The Kubernetes OOM guide covers pod-level diagnosis.

Node and RAG workers

Node’s JavaScript heap is not the same as total process memory. Large buffers, native libraries and streamed files can consume memory outside the heap.

Prefer bounded streams and queues:

for await (const chunk of documentStream) {
  await processChunk(chunk);
}

Do not accumulate every embedding or model response in one array. Limit worker concurrency and record queue depth. Raising the heap can postpone a leak while increasing the eventual blast radius.

Prevent recurrence

Track:

• resident memory and GPU memory by process;
• context length and concurrency;
• model and quantization version;
• container restarts and OOM kills;
• ingestion batch size and queue depth;
• first-token latency after memory pressure.

Alert before hard exhaustion and keep enough headroom for deployment overlap and traffic bursts. An AI memory fix is complete only when the team knows which dimension caused the failure and has a limit or test preventing it from returning.