Building a local AI agent platform

I built a local AI environment around a high-memory professional NVIDIA GPU so I could operate model serving, APIs, gateways, and agent integrations myself.

Context

The environment combines Ubuntu, NVIDIA drivers, CUDA, several inference engines, OpenAI-compatible APIs, Proxmox-hosted components, Tailscale, OpenClaw, Hermes, and local and hosted model providers.

My role

I built and operated the workstation, model servers, network access, agent gateways, service management, and model-routing experiments.

Result

I created a private platform where local applications and agents can use controlled model endpoints and where I can compare models, runtimes, context, memory use, and task fit directly.

Safe working sequence

How I reduced risk

Define boundary
Authenticate
Route
Serve
Observe
Tune

Conceptual workflow only. No live topology, identifiers, or operational data are shown.

The problem

Why the work needed more than one tool

Running a model is not the same as operating a dependable AI service. Model format, quantization, context length, VRAM, API behavior, authentication, network scope, startup, logs, storage, and client compatibility all affect the system.

What could go wrong

Exposing unauthenticated model APIs beyond the intended network
Letting experiments compete for GPU memory without clear limits
Assuming OpenAI-compatible means every client behaves identically
Tying agents to one model or provider with no operational fallback
Treating agent credentials and conversation data as harmless test material

What I did

The working sequence

Built the workstation on Ubuntu with NVIDIA drivers and CUDA
Ran models through LM Studio, Ollama, llama.cpp, and vLLM
Published OpenAI-compatible endpoints for controlled LAN and Tailscale use
Deployed OpenClaw Gateway in a Proxmox Ubuntu container and connected it to local models
Configured aliases, context windows, API authentication, agent credentials, and SSH tunneling
Connected Hermes to local and hosted providers, Telegram, Tailscale, and systemd
Separated general-task and coding models and evaluated routing by workload
Compared memory use, context behavior, inference engines, and quantization formats

How I validated it

Evidence before completion

Confirmed models loaded within available VRAM and served repeatable API responses
Tested client compatibility through real applications and agent gateways
Verified access remained limited to intended LAN or Tailscale paths
Checked service startup and recovery under systemd or container supervision
Compared model behavior on the task types the routing was intended to serve

What this demonstrates

The skill behind the tools

This work shows how I approach AI as an operational system: hardware, serving, APIs, identity, networks, clients, data boundaries, and recovery all matter.

Production details are intentionally anonymized. I do not publish organization names, internal addresses, credentials, private topology, or sensitive recovery information.

Platforms and tools

High-memory professional NVIDIA GPU Ubuntu CUDA vLLM llama.cpp LM Studio Ollama OpenClaw Hermes Proxmox Tailscale

Continue through the project inventory

All projects Experience