~/night-wave
~/night-wave
cd ~/signals
>Publishedby Shawn

Best Local AI Models for a 16GB GPU

The best local models for assistants, coding, writing, image generation, and video on a GPU with 16GB of VRAM.

#local-ai#ai-models#ollama#16gb-gpu

A modern 16GB graphics card is a versatile local AI platform, with enough VRAM for capable models across a wide range of language, multimodal, and media-generation workloads. This guide focuses on what fits within a 16GB VRAM budget, making its recommendations useful across compatible 16GB GPUs rather than tying them to one card’s compute performance.

I selected these models for their capabilities, runtime compatibility, quantization, context limits, and memory behavior. The guide explains the practical choice for each role; it does not rank models by standardized speed or quality benchmarks.

The reported VRAM percentages come from my GeForce RTX 5070 Ti with 16GB of VRAM reference setup. Capacity on other 16GB GPUs depends on architecture, runtime support, and memory overhead; generation speed can also differ.

Role Recommendation Context VRAM Usage
General I Gemma 4 12B IT (Q5_K_M) 256K 71.99%
General II Gemma 4 12B IT (Q8_0) 128K 89.08%
General III Gemma 4 26B-A4B IT (UD-Q3_K_M) 64K 91.92%
Coding I KAT-Coder V2.5 Dev REAP-50 (Q4_K_M) 256K 89.30%
Coding II KAT-Coder V2.5 Dev REAP-50 (Q5_K_M) 128K 89.25%
Coding III Qwen3.8-27B GSQ-RCO (IQ3_S) 64K 86.60%
Quick Qwen3.5 4B (Q8_0) 64K 45.59%
Agent QwenPaw-Flash 9B (Q8_0) 256K 86.70%
Creative Astrea R8 Chat 9B (Q8_0) 256K 86.70%
Image HiDream-O1-Image Full BF16 Workflow Dynamic
Video MiniMax H3 Workflow Dynamic

All nine language-model configurations stayed below 95% VRAM usage at the listed contexts in my reference checks. All three Coding tiers also have verified 100% GPU residency. VRAM usage includes memory used outside the Ollama runner.

Model Selection Criteria

  • Runtime compatibility: Each language model must load and run correctly through its selected runtime at the listed context. The KAT coding tiers have an additional runtime requirement covered under Model Aliases.

  • Claude Code compatibility: Models used with Claude Code must work at 64K or greater, natively or through a compatibility wrapper.

  • GPU residency: Language models must remain fully GPU-resident. CPU or system-RAM layer spillover does not qualify.

  • Memory fit: Language models must use 95% or less of VRAM capacity at their listed context, leaving headroom for the system and light GPU use. Image and video selections may use dynamic loading and system RAM.

  • Model quality: Selection balances capability, responsiveness, feature support, and VRAM headroom instead of choosing the largest model that can load.

Model Details

General I — Gemma 4 12B IT (Q5_K_M)

  • Context: 256K — 262,144 tokens
  • Verified Residency: 100% GPU — 49/49 layers
  • VRAM Usage: 71.99%

Gemma 4 12B IT is a dense, instruction-tuned model from Google DeepMind’s Gemma 4 family, based on the Gemma 4 12B pretrained model. It combines general reasoning with multimodal understanding. This tier uses Q5_K_M quantization, primarily 5-bit weights, at the full native context. Model card (opens in a new tab).

I chose it for General I because it pairs broad assistant capabilities with the largest context in the General stack. Q5_K_M leaves room for long conversations, multiple documents, and a large working set while keeping VRAM use comfortably below the 95% ceiling.

General II — Gemma 4 12B IT (Q8_0)

General II uses the same model as General I with Q8_0 8-bit quantization and half the native context window. Model card (opens in a new tab).

I chose it for General II because it gives everyday assistance and document work a higher-precision option when the full 256K window is unnecessary. The 128K configuration balances a substantial working context with less weight compression, making it a useful middle tier for conversations, analysis, and general problem solving.

General III — Gemma 4 26B-A4B IT (UD-Q3_K_M)

  • Context: 64K — 65,536 tokens
  • Verified Residency: 100% GPU — 31/31 layers
  • VRAM Usage: 91.92%

Gemma 4 26B-A4B IT is an instruction-tuned mixture-of-experts (MoE) model from Google DeepMind’s Gemma 4 family, based on Gemma 4 26B-A4B. It has roughly 26B total parameters with about 4B active per token. This tier uses Unsloth Dynamic UD-Q3_K_M quantization, primarily 3-bit weights with selected tensors at higher precision, and a context below its native 256K window. Model card (opens in a new tab).

I chose it for General III because its larger pool of expert parameters brings a different capability profile to focused reasoning and shorter conversations, while activating only a subset per token. The compressed weights and smaller context make that larger model practical within the 16GB budget, giving the General stack an option beyond the two dense 12B tiers.

Coding I — KAT-Coder V2.5 Dev REAP-50 (Q4_K_M)

KAT-Coder V2.5 Dev REAP-50 is an expert-pruned MoE coding model in Kwaipilot’s KAT-Coder family, whose Dev model is post-trained from Qwen3.6-35B-A3B. REAP-50 removes half the routed experts, leaving roughly 19B total parameters and about 3B active per token. This tier uses Q4_K_M quantization, primarily 4-bit weights. Base model (opens in a new tab), pruned release (opens in a new tab).

I chose it for Coding I because KAT’s training targets agentic software engineering, and the smaller Q4_K_M build leaves more room for repository context. That combination suits repository-wide analysis, multi-file changes, and extended coding sessions; the publisher also recommends this quantization for 16GB GPUs.

Coding II — KAT-Coder V2.5 Dev REAP-50 (Q5_K_M)

Coding II uses the same pruned model as Coding I with Q5_K_M quantization, primarily 5-bit weights, and half the context.

I chose it for Coding II because it preserves more weight precision for implementation, debugging, and code review while retaining a sizeable working context. The publisher identifies Q5_K_M as the higher-quality option with a tighter KV-cache budget, making this tier a good fit when the task fits within 128K tokens.

Coding III — Qwen3.8-27B GSQ-RCO (IQ3_S)

Qwen3.8-27B GSQ-RCO is a quantized release of Alibaba’s dense, 27B-parameter Qwen3.8 model, built on Qwen3.5’s architectural foundation. GSQ-RCO allocates precision by tensor sensitivity; the selected IQ3_S GGUF averages about 3.5 bits per weight. The configured context is below the base model’s native 256K window. Base model (opens in a new tab), quantized release (opens in a new tab).

I chose it for Coding III because it brings Qwen3.8’s coding and reasoning capabilities to focused changes and individual components within the 16GB budget. The publisher’s IQ3_S evaluation matched the BF16 base on LiveCodeBench v6, supporting this build as a way to retain coding capability under substantial compression. Its dense architecture also gives the stack an alternative to the two pruned KAT MoE tiers.

Quick — Qwen3.5 4B (Q8_0)

Qwen3.5 4B is a dense, 4B-parameter multimodal instruction model from Alibaba’s Qwen3.5 family, with vision, structured-output, and tool-use support. This selection uses Q8_0 8-bit quantization with a context below the model’s native 256K window. Model card (opens in a new tab).

I chose it for Quick because its compact size keeps everyday interactions responsive while Q8_0 preserves more weight precision than smaller quantizations. It handles greetings, basic Q&A, classification, application assistance, and lightweight tool calls with ample VRAM headroom.

Agent — QwenPaw-Flash 9B (Q8_0)

QwenPaw-Flash 9B is a dense, 9B-parameter model from AgentScope’s QwenPaw-Flash family, fine-tuned from Alibaba’s Qwen3.5-9B for autonomous agent tasks. This selection uses Q8_0 8-bit quantization at the native context. Its architecture includes a vision encoder, but the selected GGUF repository lists no separate vision projector. Model card (opens in a new tab).

I chose it for Agent because its training emphasizes tool invocation, command execution, file operations, information search, memory management, and multi-step planning. That specialization matches workflows that must act across several steps, while the long context provides room for instructions, tool results, and task history and Q8_0 preserves weight precision within the VRAM budget.

Creative — Astrea R8 Chat 9B (Q8_0)

Astrea R8 Chat 9B is a dense, 9B-parameter chat model from Altworld’s Astrea family, fine-tuned from Alibaba’s Qwen3.5-9B through alternating conversational and narrative training rounds. This selection uses Q8_0 8-bit quantization at the base architecture’s declared context window. Model card (opens in a new tab).

I chose it for Creative because its conversational and narrative tuning suits fiction, dialogue, prose, roleplay, and worldbuilding. The long context leaves room for character notes, setting details, prior scenes, and story constraints, while Q8_0 limits weight compression for a role where voice and phrasing matter.

Image — HiDream-O1-Image Full BF16

HiDream-O1-Image Full is an 8B-parameter image model from HiDream-ai’s O1 family, built on a dense Pixel-level Unified Transformer initialized from Qwen3-VL-8B-Instruct. The selected BF16 checkpoint uses 16-bit floating-point weights rather than a low-bit quantization. Its context combines text and reference images within the workflow; this guide does not assign it an LLM-style token window. It supports output up to 2048×2048. Technical report (opens in a new tab), ComfyUI workflow (opens in a new tab).

I chose it for Image because one model covers text-to-image generation, instruction-based editing, subject references, storyboards, and text-heavy layouts. That range makes it a versatile image workstation choice, with the Full release favored for editing and BF16 preserving weight precision. ComfyUI’s dynamic loading and the reference setup’s 64GB system RAM support this workflow within the 16GB VRAM budget.

With 32GB of system RAM, use HiDream-O1-Image Full MXFP8 instead.

Video — MiniMax H3

MiniMax H3 is an audio-video generation model from MiniMax’s H3 family. Its dense, 33B-parameter H3-Omni-Transformer generates video and audio together, while its input encoder uses pretrained Qwen3-VL-32B weights. ComfyUI provides BF16, FP8, and INT8 checkpoint variants, so precision depends on the selected workflow. Context consists of prompts and media references rather than a configured chat token window; the open weights generate clips up to about 15 seconds at 24fps. Model card (opens in a new tab), ComfyUI workflows (opens in a new tab).

I chose it for Video because its workflows combine text-to-video, image-to-video, and reference-driven generation with synchronized stereo audio. Motion control, visual conditioning, and native dialogue, sound effects, and music make it useful for complete short scenes. Quantized checkpoints and dynamic loading provide the flexibility needed for this heavier workflow on the reference 16GB GPU and 64GB system-RAM setup.

With 32GB of system RAM, use Wan2.2 TI2V 5B FP16 instead.

Stack at a Glance

My reference setup hosted Ollama and ComfyUI on a dedicated AI server, with Claude Code on another PC on the same local network. Everything can also run on one machine.

Component Configuration
Reference GPU GeForce RTX 5070 Ti — 16GB VRAM
CPU / RAM Ryzen 5 7600 — 64GB RAM
OS Ubuntu
GPU Container Runtime NVIDIA Container Toolkit
LLM Runtime Ollama in Docker
Ollama Memory Controls Flash Attention / Q8 KV cache / 1 request / 1 loaded model
Coding Client Claude Code via Ollama’s Anthropic-compatible API
Image / Video Runtime ComfyUI

Ollama Setup

For the reference NVIDIA setup, keep Docker, Ollama, and the NVIDIA driver current, confirm that nvidia-smi lists your GPU, and install the NVIDIA Container Toolkit (opens in a new tab). Ollama’s Docker guide (opens in a new tab) documents the --gpus all setup. Other GPU platforms require their corresponding Ollama and container-runtime support.

Docker Configuration

This container configuration lets each model alias define its own context:

docker run -d \
  --name ollama \
  --restart unless-stopped \
  --gpus all \
  -p 11434:11434 \
  -e OLLAMA_HOST=0.0.0.0:11434 \
  -e OLLAMA_FLASH_ATTENTION=1 \
  -e OLLAMA_KV_CACHE_TYPE=q8_0 \
  -e OLLAMA_NUM_PARALLEL=1 \
  -e OLLAMA_MAX_LOADED_MODELS=1 \
  -v ollama_data:/root/.ollama \
  ollama/ollama:latest

Flash Attention, Q8 KV cache, one parallel request, and one loaded model define the reference VRAM profile; changing them requires rechecking the 95% ceiling.

Model Aliases

Modelfiles create named configurations from the source models. Create the aliases on the Ollama server, where prompts are rendered and reasoning and tool calls are parsed. Running Claude Code on a laptop or another network PC does not change these settings. The table maps each role to its context and wrapper.

An alias created with FROM an existing Ollama model inherits its renderer/parser unless you override them. Ollama’s inheritance implementation (opens in a new tab).

Role Context / num_ctx Renderer / Parser Alias Configuration
General I 256K / 262144 gemma4 / gemma4 Inherit correct Gemma settings or configure them explicitly
General II 128K / 131072 gemma4 / gemma4 Inherit correct Gemma settings or configure them explicitly
General III 64K / 65536 gemma4 / gemma4 Inherit correct Gemma settings or configure them explicitly
Coding I 256K / 262144 qwen3.5 / qwen3.5 Shared Qwen3.5 wrapper, once runtime support is confirmed
Coding II 128K / 131072 qwen3.5 / qwen3.5 Shared Qwen3.5 wrapper, once runtime support is confirmed
Coding III 64K / 65536 qwen3.8 / qwen3.5 Qwen3.8-specific wrapper
Quick 64K / 65536 qwen3.5 / qwen3.5 Shared Qwen3.5 wrapper
Agent 256K / 262144 qwen3.5 / qwen3.5 Shared Qwen3.5 wrapper
Creative 256K / 262144 qwen3.5 / qwen3.5 Shared Qwen3.5 wrapper

Image and Video use ComfyUI workflows and do not need Ollama aliases. The renderer/parser names above are supported in Ollama’s current source; the installed server must support both the selected names and the model architecture. Renderer registry (opens in a new tab), parser registry (opens in a new tab).

Gemma 4 Settings — General I / II / III

Use the selected Gemma source and the num_ctx from the table. For a standalone GGUF import, explicitly configure the native Gemma 4 renderer and parser. This example uses General I’s context:

FROM SOURCE_MODEL
TEMPLATE {{ .Prompt }}
RENDERER gemma4
PARSER gemma4
PARAMETER num_ctx 262144

An alias based on a complete official Ollama Gemma 4 package can inherit its existing Gemma settings and projector with only FROM and PARAMETER num_ctx. Inspect the source first and preserve any checkpoint-specific Gemma renderer variant. All three General tiers use Gemma handling; do not apply a Qwen wrapper to them.

Shared Qwen3.5 Wrapper — Coding I / II, Quick, Agent, and Creative

Use this block only for the roles in the heading, taking the source from Model Details and num_ctx from the table. The example uses a 256K context:

FROM SOURCE_MODEL
TEMPLATE {{ .Prompt }}
RENDERER qwen3.5
PARSER qwen3.5
PARAMETER num_ctx 262144

Set the Qwen renderer/parser explicitly for standalone GGUF imports rather than relying on autodetection. These directives configure prompt and output handling; they do not add a vision projector.

Creative caveat: Astrea uses Qwen3.5 architecture, but its community GGUF has a customized non-reasoning chat template. The wrapper above is the architecture-compatible choice for Claude Code; verify its behavior after overriding that template. Creative-writing tuning alone does not establish reliable tool-agent behavior. GGUF release (opens in a new tab).

Coding I / II runtime requirement: The pruned KAT release retains the qwen35moe architecture and Qwen3.5-compatible handling. Its publisher documents a recent llama.cpp build and lists Ollama support as pending. These alias settings apply only once the installed Ollama runtime can load the model; a renderer/parser override does not add architecture support. Pruned model card (opens in a new tab).

Qwen3.8 Wrapper — Coding III

Coding III uses a distinct prompt renderer with the Qwen3.5 output parser:

FROM hf.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S
TEMPLATE {{ .Prompt }}
RENDERER qwen3.8
PARSER qwen3.5
PARAMETER num_ctx 65536

Create and Inspect Aliases

SOURCE_MODEL is the source model or GGUF path listed under Model Details. The alias name passed to ollama create is what the client selects. For example, save the shared Qwen3.5 block with FROM qwen3.5:4b-q8_0 and PARAMETER num_ctx 65536 as Modelfile.quick and create Quick’s documented alias:

ollama create qwen3.5:4b-q8_0-64k -f Modelfile.quick
ollama show qwen3.5:4b-q8_0-64k
ollama show qwen3.5:4b-q8_0-64k --modelfile

Run ollama show SOURCE_MODEL --modelfile before creating an alias, then run both ollama show MODEL_NAME and ollama show MODEL_NAME --modelfile afterward. Check the alias’s capabilities, renderer, parser, and num_ctx. With the Docker setup, use docker exec ollama ollama show --modelfile MODEL_NAME on the server. Other alias names are your choice; keep them consistent with the launcher’s menu. Before relying on an alias in Claude Code, verify a tool call through the Anthropic-compatible endpoint as well as ordinary chat. A successful model load alone does not verify tool-call handling. Modelfile reference (opens in a new tab).

Claude Code Setup

Claude Code connects through Ollama’s Anthropic-compatible API (opens in a new tab). It selects the server-side aliases configured above, whether it runs on the server or another computer.

Local AI Launcher

The exact launcher is personal preference, but every local Claude Code session needs to:

  • Point ANTHROPIC_BASE_URL at Ollama’s Anthropic-compatible endpoint.

  • Set ANTHROPIC_AUTH_TOKEN and define ANTHROPIC_API_KEY; the API key can be empty because Ollama does not use it as a credential.

  • Select an Ollama alias and set CLAUDE_CODE_MAX_CONTEXT_TOKENS to its configured context.

My setup packages this into a user-local custom-claude-code script. It presents the recommended aliases in a menu, exports the required values, and launches Claude Code with the selection. Other shell scripts, aliases, or launchers can implement the same requirements.

Vision Settings by Role

Configure Claude Code for the capabilities of the actual Ollama alias, rather than the upstream model architecture. A vision-capable architecture still needs its matching vision projector in the deployed package. A standalone text GGUF without that projector should be treated as text-only.

Role Claude Code vision setting
General I / II / III NO for a text GGUF alone; YES if the alias contains or inherits the matching projector
Coding I / II NO — the selected REAP-50 GGUF release strips the vision tower
Coding III NO for the IQ3_S text GGUF alone; YES only with its matching projector
Quick YES when inheriting the complete qwen3.5:4b-q8_0 package; NO for a standalone text GGUF without its projector
Agent NO for the selected Q8_0 GGUF deployment, which lists no separate projector
Creative NO for the selected Astrea Q8_0 GGUF deployment; keep this slot text-only

The KAT release explicitly strips vision before producing the GGUF builds. Quick’s official Ollama package advertises text and image input, so an alias created from that complete package retains vision. KAT build details (opens in a new tab), Quick package (opens in a new tab).

On the Ollama server, run these checks for each alias:

ollama show MODEL_NAME
ollama show MODEL_NAME --modelfile

With the Docker setup, prefix each command with docker exec ollama. If the alias’s capabilities list includes vision, enable image input for the conditional roles above and verify an image request. If vision is absent, configure that alias as NO vision, even if its upstream architecture supports images. Inspect the Modelfile separately to confirm the intended renderer/parser survived creation.

For every text-only selection—including both KAT Coding tiers, Agent, Creative, and any General, Coding III, or Quick alias without a projector—the launcher should block visual inputs and append a matching system instruction. A system instruction alone does not remove image attachments or add multimodal support. If all nine aliases were imported from standalone text GGUFs without projectors, configure NO vision for all nine.

Adding a projector changes memory use. Recheck GPU residency and the 95% VRAM ceiling at the configured context before treating a newly multimodal alias as equivalent to the reference configuration.

Linux Desktop Shortcut

The custom-claude-code-here helper accepts the %k launcher field code, removes a possible file:// prefix, changes to the launcher’s directory, and starts the main script:

#!/usr/bin/env bash

launcher="$1"
launcher="${launcher#file://}"

cd -- "$(dirname -- "$launcher")" || exit 1
exec "$HOME/.local/bin/custom-claude-code"

Create a desktop launcher for quick access to the custom Claude Code setup:

/home/USERNAME/.local/bin/custom-claude-code-here %k

ComfyUI Setup

Install or update ComfyUI (opens in a new tab), then use Workflow → Browse Templates. The Image and Video model details link to the official workflows and required files.

Before generating, close other GPU-heavy applications and unload Ollama models with docker exec ollama ollama stop MODEL_ID. Start with batch size 1 and a 1024px image or short 720p clip, then scale up after confirming a stable run.