Vision models can look at an image and tell you what’s in it — describe a photo, read text from a screenshot, or generate alt text for accessibility. The best part? You can run these models entirely on your own machine with Ollama.

In this tutorial, we’ll build a Python tool that takes any image file, sends it to a local vision model, and returns a description, alt text, or extracted text. We’ll wire up both a CLI and a minimal web UI so you can use it however you prefer.

No API keys. No cloud. Just your GPU (or CPU) and a vision model.

What You’ll Build

• A Python script that sends images to Ollama vision models (LLaVA, BakLLaVA, or Moondream)
• Three modes: describe (general description), alt (short alt text), and ocr (extract text from screenshots)
• A CLI for quick terminal use
• A Flask web UI for drag-and-drop convenience

Prerequisites

• Python 3.10+
• Ollama installed and running
• A vision model pulled locally

Pull a vision model before starting. LLaVA is the most capable general-purpose option, while Moondream is lighter if you’re short on VRAM:

ollama pull llava
# or for lower resource usage:
ollama pull moondream

For a deeper look at which models work best for different local tasks, see the best AI models for coding and tasks locally guide.

Install the Python dependencies:

pip install requests flask

Project Structure

image-describer/
├── describer.py      # Core logic + CLI
├── web.py            # Flask web UI
└── templates/
    └── index.html    # Upload form

Core Logic — describer.py

This module handles encoding the image and calling the Ollama API. It also doubles as the CLI entry point.

import argparse
import base64
import json
import sys
from pathlib import Path

import requests

OLLAMA_URL = "http://localhost:11434/api/generate"

PROMPTS = {
    "describe": "Describe this image in detail.",
    "alt": "Write a short, concise alt text for this image (one sentence).",
    "ocr": "Extract all visible text from this image. Return only the text, nothing else.",
}


def describe_image(image_path: str, mode: str = "describe", model: str = "llava") -> str:
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Image not found: {image_path}")

    image_b64 = base64.b64encode(path.read_bytes()).decode()

    resp = requests.post(OLLAMA_URL, json={
        "model": model,
        "prompt": PROMPTS.get(mode, PROMPTS["describe"]),
        "images": [image_b64],
        "stream": False,
    }, timeout=120)
    resp.raise_for_status()
    return resp.json()["response"].strip()


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Describe images with local AI vision models")
    parser.add_argument("image", help="Path to image file")
    parser.add_argument("-m", "--mode", choices=PROMPTS.keys(), default="describe",
                        help="describe | alt | ocr (default: describe)")
    parser.add_argument("--model", default="llava", help="Ollama vision model (default: llava)")
    args = parser.parse_args()

    try:
        result = describe_image(args.image, args.mode, args.model)
        print(result)
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        sys.exit(1)

The key piece is the /api/generate endpoint with the images array — Ollama expects base64-encoded image data alongside the prompt. Setting stream: False gives us the full response in one shot instead of token-by-token.

Using the CLI

# General description
python describer.py photo.jpg

# Generate alt text
python describer.py screenshot.png -m alt

# Extract text from a screenshot
python describer.py receipt.png -m ocr

# Use a different model
python describer.py photo.jpg --model moondream

Web UI — web.py

A small Flask app that lets you upload an image through the browser and see the result.

from flask import Flask, render_template, request
from describer import describe_image, PROMPTS
import tempfile
from pathlib import Path

app = Flask(__name__)


@app.route("/", methods=["GET", "POST"])
def index():
    result = None
    error = None
    if request.method == "POST":
        file = request.files.get("image")
        mode = request.form.get("mode", "describe")
        model = request.form.get("model", "llava")
        if not file or not file.filename:
            error = "Please select an image."
        else:
            try:
                suffix = Path(file.filename).suffix
                with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
                    file.save(tmp.name)
                    result = describe_image(tmp.name, mode, model)
            except Exception as e:
                error = str(e)
    return render_template("index.html", result=result, error=error, modes=PROMPTS.keys())


if __name__ == "__main__":
    app.run(debug=True, port=5000)

HTML Template — templates/index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>AI Image Describer</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 600px; margin: 2rem auto; padding: 0 1rem; }
    select, input, button { padding: 0.5rem; margin: 0.25rem 0; }
    button { cursor: pointer; background: #2563eb; color: white; border: none; border-radius: 4px; }
    .result { background: #f1f5f9; padding: 1rem; border-radius: 6px; margin-top: 1rem; white-space: pre-wrap; }
    .error { color: #dc2626; }
  </style>
</head>
<body>
  <h1>🖼️ AI Image Describer</h1>
  <form method="POST" enctype="multipart/form-data">
    <div>
      <label>Image: <input type="file" name="image" accept="image/*" required></label>
    </div>
    <div>
      <label>Mode:
        <select name="mode">
          {% for mode in modes %}<option value="{{ mode }}">{{ mode }}</option>{% endfor %}
        </select>
      </label>
    </div>
    <div>
      <label>Model: <input type="text" name="model" value="llava"></label>
    </div>
    <button type="submit">Describe</button>
  </form>
  {% if error %}<p class="error">{{ error }}</p>{% endif %}
  {% if result %}<div class="result">{{ result }}</div>{% endif %}
</body>
</html>

Run the web UI:

python web.py

Open http://localhost:5000, upload an image, pick a mode, and hit Describe.

Choosing a Vision Model

Model	Size	Best For	Speed
llava	~4.7 GB	General descriptions, detailed analysis	Medium
bakllava	~4.7 GB	Similar to LLaVA, slightly different style	Medium
moondream	~1.7 GB	Quick descriptions, lower resource usage	Fast

LLaVA gives the most detailed and accurate results for most tasks. Moondream is a solid pick when you need speed or have limited hardware. Try both and see which fits your workflow — you can swap models with a single flag.

Want to try this without buying hardware? Cloud GPU providers let you spin up the right GPU in minutes.

For more on choosing the right model for local tasks, the Qwen 3 guide covers the latest options in the small-model space.

Practical Use Cases

• Accessibility: Generate alt text for images on your website or in documents. Run the tool in alt mode across a folder of images with a quick shell loop:

for img in images/*.jpg; do
  echo "$img: $(python describer.py "$img" -m alt)"
done

• Screenshot OCR: Extract text from screenshots, error messages, or photos of documents without uploading anything to the cloud.

• Content cataloging: Describe product photos, artwork, or archival images for search indexing.

• Development workflows: Integrate into CI/CD pipelines to auto-generate alt text for static site images, similar to how we integrated Ollama in the AI translation tool tutorial.

Tips for Better Results

• Image size matters: Vision models work best with clear, reasonably sized images. Extremely large files slow things down — resize to ~1024px on the longest side if speed is an issue.
• Be specific in prompts: You can edit the PROMPTS dict to tailor output. For example, change the alt prompt to "Write alt text for this image, under 125 characters." for stricter length control.
• GPU acceleration: If Ollama detects a compatible GPU, vision models run significantly faster. Check with ollama ps to confirm your model is using GPU layers.

What’s Next

You now have a fully local image description tool — no API keys, no data leaving your machine. From here you could:

• Add batch processing to handle entire directories
• Pipe output into a static site generator to auto-fill alt attributes
• Combine with the translation tool to describe images in multiple languages
• Build a Raycast/Alfred extension that describes your clipboard image

The vision model ecosystem is moving fast. Keep an eye on new models via ollama list and the Ollama complete guide for updates.