Skip to content
Evonic AI

Ollama Setup

Ollama is the easiest way to run large language models locally. It provides a simple CLI and API interface with support for many popular models.

Installation

Section titled “Installation”

macOS

Section titled “macOS”
Terminal window
brew install ollama

Linux

Section titled “Linux”
Terminal window
curl -fsSL https://ollama.com/install.sh | sh

Windows

Section titled “Windows”

Download the installer from ollama.com

Getting Started

Section titled “Getting Started”

1. Start Ollama

Section titled “1. Start Ollama”
Terminal window
ollama serve

2. Pull a Model

Section titled “2. Pull a Model”
Terminal window
ollama pull llama3.2
ollama pull mistral
ollama pull codellama

3. Test the Model

Section titled “3. Test the Model”
Terminal window
ollama run llama3.2

Configuration with Evonic AI

Section titled “Configuration with Evonic AI”

API Endpoint

Section titled “API Endpoint”

Ollama runs a local API server by default at http://localhost:11434.

Configuration

Section titled “Configuration”

In your Evonic AI configuration:

model:
  provider: ollama
  endpoint: http://localhost:11434
  model_name: llama3.2

Available Models

Section titled “Available Models”

Browse available models at ollama.com/library or list them:

Terminal window
ollama list

Model Management

Section titled “Model Management”

List Installed Models

Section titled “List Installed Models”
Terminal window
ollama list

Remove a Model

Section titled “Remove a Model”
Terminal window
ollama rm llama3.2

Copy a Model

Section titled “Copy a Model”
Terminal window
ollama cp llama3.2 llama3.2-custom

Advanced Configuration

Section titled “Advanced Configuration”

GPU Acceleration

Section titled “GPU Acceleration”

Ollama automatically uses GPU if available. To force CPU-only:

Terminal window
OLLAMA_NUM_GPU=0 ollama serve

Custom Port

Section titled “Custom Port”
Terminal window
OLLAMA_HOST=0.0.0.0:11435 ollama serve

Model Parameters

Section titled “Model Parameters”

Create a Modelfile for custom parameters:

FROM llama3.2
PARAMETER temperature 0.7
PARAMETER num_ctx 4096
SYSTEM """You are a helpful assistant."""

Build the custom model:

Terminal window
ollama create my-model -f Modelfile

Troubleshooting

Section titled “Troubleshooting”

Model Not Loading

Section titled “Model Not Loading”
  • Check GPU memory availability
  • Try a smaller model (e.g., llama3.2:1b instead of llama3.2:70b)
  • Ensure Ollama is running: ollama list

Slow Inference

Section titled “Slow Inference”
  • Enable GPU acceleration
  • Reduce num_ctx parameter
  • Close other GPU-intensive applications

Connection Refused

Section titled “Connection Refused”
  • Verify Ollama is running: curl http://localhost:11434
  • Check firewall settings
  • Ensure correct endpoint in configuration