Skip to content

Knowledge base

AI Models and Inference

Which model runs an agent, where it runs and what each token costs determine whether an agentic workflow pays off. Outcomes depend more on the harness around the model, its context management, tool selection and validation, than on which frontier model sits underneath.

Written by

Updated:

Tools 4

Our position. We choose the model per station, not per project. Planning runs on the strongest model we have, and implementation runs on a cheaper, faster one that executes a plan already reviewed.

The model layer spans frontier and open-weight models, cloud inference providers and local inference on consumer hardware, each with its own price per token. Coding benchmarks help rank candidates, and schema libraries make model output safe for code to consume.

Start here

Resources

Cloud Models & Providers

Gateways & Routing

  • LiteLLM (opens in a new tab) - Open-source AI gateway and SDK giving unified OpenAI-format access to Azure, Bedrock, Gemini, OpenAI, and Anthropic with routing and spend tracking.
  • OpenRouter (opens in a new tab) - Unified API gateway that routes requests across hundreds of models from many providers with usage-based billing and fallbacks.
  • Portkey (opens in a new tab) - Production gateway for GenAI apps offering routing, caching, guardrails, and observability across providers; being acquired by Palo Alto Networks.
  • Weave Router (opens in a new tab) - Prompt router that sends each request to the best quality-per-token model inside Codex, Claude, and Cursor to stretch token budgets.

Local Inference

  • Atomic Chat (opens in a new tab) - (atomic.chat (opens in a new tab)) Open-source ChatGPT alternative built with Tauri for macOS that runs local HuggingFace models or connects to OpenAI, Anthropic, and others, with an OpenAI-compatible local server.
  • Colibri (opens in a new tab) - Pure-C engine (Apache 2.0) that runs the 744B GLM-5.2 MoE on ~25GB RAM by streaming routed experts from disk on demand.
  • Jan (opens in a new tab) - (jan.ai (opens in a new tab)) Open-source ChatGPT alternative (Tauri, Windows/macOS/Linux, also on Microsoft Store and Flathub) that runs local LLMs via llama.cpp and connects to remote providers.
  • LibreChat (opens in a new tab) - Open-source, self-hostable AI chat platform with agents, code interpreter, and artifacts, connecting to Anthropic, OpenAI, Azure, AWS, and local models behind one interface.
  • Lucebox (opens in a new tab) - Hub of hand-tuned LLM inference projects for specific consumer hardware (RTX 3090/4090/5090, DGX Spark, Jetson AGX Thor) with per-chip kernel rewrites, full benchmarks, and docs.
  • MTPLX (opens in a new tab) - (mtplx.com (opens in a new tab)) MLX-native inference server for Apple Silicon that runs the model’s own MTP heads as a speculative drafter, hitting ~2.24x decode speedup, with OpenAI-compatible endpoints.
  • ODS (opens in a new tab) - One-command Docker installer that turns any machine into a private AI server, wiring Ollama, Open WebUI, n8n, ComfyUI, and Whisper together; Apache 2.0.
  • TurboQuant+ (opens in a new tab) - Implementation of TurboQuant (ICLR 2026) for KV-cache compression in local LLM inference, hitting 3.8-6.4x compression, tested up to 104B models at 128K context.
  • llama.cpp TurboQuant (opens in a new tab) - Fork of llama.cpp adding TurboQuant quantization for local LLM inference in C/C++.
  • omlx (opens in a new tab) - (omlx.ai (opens in a new tab)) macOS menu-bar inference server for Apple Silicon built on MLX, with an OpenAI-compatible API, continuous batching, and SSD caching to serve multiple local models concurrently.
  • vLLM Swift (opens in a new tab) - vLLM Metal plugin powered by mlx-swift for high-performance LLM inference on Apple Silicon.
  • whichllm (opens in a new tab) - CLI that ranks local LLMs for your hardware using recency-aware benchmarks rather than parameter count, estimating VRAM and inference speed with confidence-graded evidence; MIT.

Model Collections

Rankings & Reports

Structured Output

  • BoundaryML (opens in a new tab) - BAML, an open-source programming language for agents with runtime types and native LLM functions, tests, and evaluations; pre-1.0.
  • Instructor (opens in a new tab) - Multi-language library (Python, TypeScript, Go, Ruby, Elixir, Rust) for extracting validated structured outputs from 15+ LLM providers including Claude and OpenAI, with Pydantic schemas, retries, and streaming.

In this section

  • Inference economics and infrastructure

    Agentic workloads make inference cost usage-shaped: parallel agents and growing context drive token consumption, so the bill depends on tokens per completed workflow more than on the list price per token. Routing simple tasks to small models and reserving frontier models for the work that needs them keeps it in proportion.

  • Choosing models

    Choose a model by testing it on your own stack and tasks, not by its leaderboard score: coding benchmarks measure narrow, Python-heavy, well-specified issues, and capability varies sharply by language and task. Routing each role to a fitting model and investing in context and validation give more leverage than chasing the top model.

  • Structured model output

    Structured output means a model answers in a typed shape that code can validate, instead of free text that code has to parse. A schema-first layer declares the shape as a typed model, checks every response against it, and re-asks the model when validation fails.