Lab 1: Run Gemma 4 (12B) Locally with Apple MLX & Token Streaming

Stop paying commercial API bills for agent reasoning loops. Build a high-performance, 100% private local inference layer with Google Gemma 4 (12B) and Apple MLX.

Download notebook to follow along: https://github.com/MysteryBytes-Labs/agentic_ai/tree/main/ai_trading_coach_Gemma4

Why Local LLMs Are Essential for Autonomous Financial Agents

Autonomous agents do not operate like conventional conversational chatbots. A single user inquiry (such as "Scan the market, purge penny stocks, and recommend three breakout opportunities") triggers an intensive chain of internal steps:

• Task decomposition into sub-goals
• Multi-turn tool argument synthesis and validation
• Dynamic web scraping and schema parsing
• Financial ratio valuation and numerical sanity checks
• Iterative reflection and critique before final memo synthesis

In an agentic architecture, a single workflow can easily require 30 to 100 round-trip LLM invocations. When routed through proprietary cloud APIs:

• Token costs compound exponentially: Every step resends conversation history, tool definitions, and system prompts.
• Latency kills responsiveness: Network round-trips over the public internet introduce 500ms to 2s of latency per loop.
• Rate limits break pipelines: Cloud provider quotas and sudden throttling disrupt autonomous multi-agent systems.
• Data privacy is compromised: Sensitive financial watchlists, portfolio balances, and proprietary trading strategies are transmitted to third-party cloud servers.

By hosting Google Gemma 4 (12B) locally with Apple MLX, you transform your Mac into an autonomous agent sandbox with zero per-token API costs, unified memory zero-copy inference, and complete data isolation.

Apple MLX Unified Memory Architecture

Apple MLX is an array framework designed specifically for Apple Silicon. Unlike traditional runtimes (which copy tensor weights back and forth across PCIe buses between host CPU RAM and discrete GPU VRAM), MLX executes computations directly on Apple Silicon's Unified Memory Architecture (UMA) via Apple Metal.

┌────────────────────────────────────────────────────────────────┐
│                   APPLE SILICON UNIFIED MEMORY                 │
│                                                                │
│   ┌───────────────────────────┐    ┌───────────────────────┐   │
│   │   CPU Cores (Python Host) │◄──►│  Metal GPU Cores      │   │
│   └───────────────────────────┘    └───────────────────────┘   │
│                 ▲                              ▲               │
│                 │      Zero-Copy Memory Bus    │               │
│                 └──────────────┬───────────────┘               │
│                                ▼                               │
│        ┌──────────────────────────────────────────────┐        │
│        │  Shared Pool (Gemma 4 12B Quantized Weights) │        │
│        └──────────────────────────────────────────────┘        │
└────────────────────────────────────────────────────────────────┘

This zero-copy architecture allows our 12-billion parameter Gemma 4 model to run at peak throughput with a compact ~10 GB local footprint—avoiding duplicate host-to-device memory copies and enabling instantaneous token generation.

Prerequisites & System Requirements

Before running the notebook, verify your machine meets the hardware and software configuration:

Step 1: Environment Verification & Apple Metal Acceleration

In your terminal or virtual environment, install the MLX packages:

pip install mlx mlx-vlm ipython

In the first cell of lab1_gemma3_setup.ipynb, import mlx.core and verify that Apple Metal acceleration is active:

import os
import mlx.core as mx

# 1. Verify Apple Metal GPU availability
print(f"MLX Version: {mx.__version__}")
print(f"Apple Metal GPU Available: {mx.metal.is_available()}")

# 2. Inspect local model directory
MODEL_PATH = "AIModels/gemma-4-12B-it-qat-4bit"
assert os.path.exists(MODEL_PATH), f"Model directory not found at {MODEL_PATH}"

print(f"Model weights verified at: {MODEL_PATH}")

When run on Apple Silicon, mx.metal.is_available() returns True, confirming that MLX will route all matrix multiplications directly to the Metal execution pipeline.

Step 2: Loading Gemma 4 into Unified Memory with mlx-vlm

Gemma 4 features a unified multimodal architecture. We load both the model weights and the tokenizer/processor via mlx_vlm.load:

from mlx_vlm import load

print("Loading Gemma 4 (12B QAT) into Unified Memory...")
model, processor = load(MODEL_PATH)

print(f"Model Type: {type(model).__name__}")
print("Gemma 4 successfully mapped to Apple Metal GPU device.")

Behind the scenes, MLX memory-maps the 4-bit quantized safetensors weights directly into unified RAM. In under three seconds, the model is initialized with zero memory duplication.

Step 3: Low-Latency Token Streaming via stream_generate

In autonomous agents, waiting for a multi-hundred-token generation to finish before receiving output introduces noticeable latency. By enabling token streaming with MLX's stream_generate generator, the engine emits text chunks the instant each token is decoded:

import sys
from mlx_vlm import stream_generate

prompt = "Write an optimized Python function to reverse a string in-place or using slicing. Explain time complexity."

print("Streaming generation:\n")
for chunk in stream_generate(model, processor, prompt=prompt, max_tokens=256):
    sys.stdout.write(chunk.text)
    sys.stdout.flush()

print("\n\n--- Stream Completed ---")

stream_generate yields decoded text chunks iteratively, allowing you to build real-time visual streaming for user interfaces or stream tokens directly into intermediate parsing routines.

Step 4: Structured Jinja Chat Templates & System Prompts

Instruction-tuned models require exact turn delimiters (<start_of_turn>, <end_of_turn>) to distinguish system instructions from user queries and model outputs. Rather than manually concatenating strings, we use processor.apply_chat_template:

conversation = [
    {
        "role": "system",
        "content": "You are a precise, data-driven financial research analyst. Provide structured tabular outputs."
    },
    {
        "role": "user",
        "content": "Compare New York City and Los Angeles across Population, Area (sq mi), and Top Financial Sector."
    }
]

# Format chat prompt using official Jinja template
formatted_prompt = processor.apply_chat_template(
    conversation,
    tokenize=False,
    add_generation_prompt=True
)

print("Formatted Jinja Prompt Structure:\n")
print(formatted_prompt)

Setting add_generation_prompt=True automatically appends the model turn prefix, cueing Gemma 4 to immediately begin generating the response.

Step 5: Full-Sequence Inference & Rich Display Formatting

To execute non-streaming generation and capture complete outputs, we use MLX's generate function with a low temperature for deterministic consistency:

from mlx_vlm import generate
from IPython.display import display, Markdown

# Generate response with deterministic temperature
response_text = generate(
    model,
    processor,
    prompt=formatted_prompt,
    max_tokens=512,
    temp=0.2
)

# Render clean markdown table in Jupyter notebook
display(Markdown(response_text))

Gemma 4 outputs a clean, well-formed Markdown comparison table complete with column headers and demographic statistics.

Key Takeaways & Series Roadmap

With Lab 1 complete, our high-speed local inference engine is verified and operational on Apple Silicon Metal:

What's Next in Lab 2?

Right now, Gemma 4 is functioning as a static conversational model. In Lab 2, we give our agent hands:

• Enforcing zero-hallucination outputs using Pydantic v2 schemas
• Dynamic Temporal Grounding anchored to the live hardware clock
• Live market telemetry tool calling via the Google Serper API