> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/block/goose/llms.txt
> Use this file to discover all available pages before exploring further.

# Local Inference

> Run Goose with local LLMs using llama.cpp for offline and privacy-focused AI

Goose supports local inference using [llama.cpp](https://github.com/ggerganov/llama.cpp), enabling offline usage, data privacy, and cost savings. This guide covers setup, configuration, and optimization.

## Overview

Local inference allows you to:

* **Run offline**: No internet required once models are downloaded
* **Preserve privacy**: Data never leaves your machine
* **Eliminate API costs**: No per-token charges
* **Customize models**: Use fine-tuned or specialized models
* **Control resources**: Manage GPU/CPU usage precisely

## Quick Start

### 1. Install Prerequisites

**macOS/Linux:**

```bash theme={null}
# llama.cpp is bundled with Goose
# Just ensure you have enough disk space for models
```

**GPU Acceleration (Optional):**

```bash theme={null}
# CUDA (NVIDIA)
sudo apt install nvidia-cuda-toolkit

# ROCm (AMD)
sudo apt install rocm-hip-runtime

# Metal (macOS)
# Built-in, no installation needed
```

### 2. Download a Model

Goose supports GGUF format models from Hugging Face:

```bash theme={null}
# Download a recommended model (Qwen 2.5 Coder)
goose download-model qwen2.5-coder-7b-instruct

# Or specify a Hugging Face repo
goose download-model Qwen/Qwen2.5-Coder-7B-Instruct-GGUF
```

Models are stored in `~/.cache/goose/models/`.

### 3. Configure Goose

```bash theme={null}
# Set provider to local
goose configure set GOOSE_PROVIDER local

# Specify model
goose configure set GOOSE_MODEL qwen2.5-coder-7b-instruct
```

### 4. Run Goose

```bash theme={null}
goose session start
```

Goose will automatically:

1. Load the model into memory
2. Allocate GPU/CPU resources
3. Initialize the inference engine

## Supported Models

Goose works with any GGUF model, but these are recommended:

### Coding Models

| Model                     | Size | Context | Best For                   |
| ------------------------- | ---- | ------- | -------------------------- |
| **Qwen 2.5 Coder 7B**     | 7B   | 32K     | General coding, fast       |
| **Qwen 2.5 Coder 14B**    | 14B  | 32K     | Complex tasks, accurate    |
| **DeepSeek Coder V2 16B** | 16B  | 128K    | Long context, architecture |
| **CodeLlama 13B**         | 13B  | 16K     | Python, legacy             |

### General Purpose

| Model            | Size | Context | Best For             |
| ---------------- | ---- | ------- | -------------------- |
| **Llama 3.1 8B** | 8B   | 128K    | Balanced performance |
| **Mistral 7B**   | 7B   | 32K     | Fast, efficient      |
| **Phi-3 Mini**   | 3.8B | 128K    | Low memory           |

<Info>
  Model sizes are approximate. Quantized versions (Q4, Q5, Q6) reduce memory usage at the cost of slight accuracy loss.
</Info>

## Quantization Levels

GGUF models come in various quantization levels:

| Quantization | Size | Quality    | Speed   | Recommended     |
| ------------ | ---- | ---------- | ------- | --------------- |
| Q8\_0        | 100% | Best       | Slow    | High VRAM       |
| Q6\_K        | 75%  | Excellent  | Medium  | Balanced        |
| Q5\_K\_M     | 65%  | Very Good  | Fast    | **Recommended** |
| Q4\_K\_M     | 50%  | Good       | Faster  | Low VRAM        |
| Q3\_K\_M     | 40%  | Acceptable | Fastest | Constrained     |

**Example:** Download specific quantization:

```bash theme={null}
goose download-model Qwen/Qwen2.5-Coder-7B-Instruct-GGUF --quant Q5_K_M
```

## Configuration

### Model Settings

Configure in `~/.config/goose/config.yaml`:

```yaml theme={null}
GOOSE_PROVIDER: local
GOOSE_MODEL: qwen2.5-coder-7b-instruct

# Local inference settings
local_inference:
  # Context size (tokens)
  context_size: 8192
  
  # Generation settings
  temperature: 0.7
  top_p: 0.95
  top_k: 40
  repeat_penalty: 1.1
  
  # Performance
  n_batch: 512
  n_threads: 8
  flash_attention: true
  
  # GPU offload (0 = CPU only, -1 = all layers)
  n_gpu_layers: -1
```

### Environment Variables

```bash theme={null}
# Force CPU inference
export GOOSE_LOCAL_CPU_ONLY=1

# Set GPU layers manually
export GOOSE_N_GPU_LAYERS=32

# Model cache directory
export GOOSE_MODEL_CACHE=~/.cache/goose/models
```

## Memory Management

Goose automatically estimates memory requirements and adjusts context size.

### Memory Estimation

From `crates/goose/src/providers/local_inference/inference_engine.rs`:

```rust theme={null}
pub fn estimate_max_context_for_memory(
    model: &LlamaModel,
    runtime: &InferenceRuntime,
) -> Option<usize> {
    let available = available_inference_memory_bytes(runtime);
    
    // Reserve 50% for computation buffers
    let usable = (available as f64 * 0.5) as u64;
    
    // Calculate KV cache size per token
    let bytes_per_token = (k_per_head + v_per_head) * n_head_kv * n_layer * 2;
    
    Some((usable / bytes_per_token) as usize)
}
```

### Memory Requirements

Typical requirements for Q5\_K\_M quantization:

| Model Size | Model RAM | Context (8K) | Context (32K) | Total    |
| ---------- | --------- | ------------ | ------------- | -------- |
| 3B         | 2.5 GB    | 0.5 GB       | 2 GB          | 3-5 GB   |
| 7B         | 5 GB      | 1 GB         | 4 GB          | 6-9 GB   |
| 13B        | 9 GB      | 2 GB         | 8 GB          | 11-17 GB |
| 34B        | 22 GB     | 5 GB         | 20 GB         | 27-42 GB |

<Warning>
  If your prompt exceeds available memory, Goose will return an error: "Prompt exceeds estimated memory capacity". Reduce context size or use a smaller model.
</Warning>

## Performance Tuning

### GPU Acceleration

**Check GPU usage:**

```bash theme={null}
# NVIDIA
nvidia-smi

# AMD
rocm-smi

# macOS
sudo powermetrics --samplers gpu_power
```

**Optimize GPU layers:**

```yaml theme={null}
# Start with all layers
n_gpu_layers: -1

# If OOM, reduce gradually
n_gpu_layers: 32  # Offload 32 layers to GPU
```

### CPU Optimization

```yaml theme={null}
# Use all CPU cores
n_threads: -1  # Auto-detect

# Or specify manually
n_threads: 16

# Batch size (larger = faster, more memory)
n_batch: 512
```

### Flash Attention

Enable for 2-3x faster inference on supported hardware:

```yaml theme={null}
flash_attention: true
```

Requires:

* CUDA compute capability ≥ 7.0 (RTX 20 series+)
* Metal (macOS M1+)
* ROCm 5.0+

## Tool Support

Local models support tool calling through two modes:

### Native Tools (Preferred)

Models trained with native tool calling (e.g., Qwen 2.5):

```rust theme={null}
// Automatically detected from model metadata
if model.supports_native_tools() {
    // Use built-in tool calling
}
```

### Emulated Tools

For models without native support, Goose emulates tool calling:

```rust theme={null}
// Inject tool definitions into system prompt
let prompt_with_tools = format!(
    "{}\n\nAvailable tools:\n{}",
    system_prompt,
    tool_definitions
);
```

<Tip>
  Native tool calling is more reliable. Choose models like Qwen 2.5, Mistral, or Llama 3.1 for best results.
</Tip>

## Sampling Strategies

### Temperature Sampling (Default)

Balanced creativity and coherence:

```yaml theme={null}
sampling:
  type: temperature
  temperature: 0.7  # 0 = deterministic, 1 = creative
  top_p: 0.95       # Nucleus sampling
  top_k: 40         # Top-k sampling
  min_p: 0.05       # Minimum probability
```

### Greedy Sampling

Always select most likely token (deterministic):

```yaml theme={null}
sampling:
  type: greedy
```

### Mirostat v2

Adaptive sampling for consistent perplexity:

```yaml theme={null}
sampling:
  type: mirostat_v2
  tau: 5.0   # Target perplexity
  eta: 0.1   # Learning rate
```

## Troubleshooting

### Model won't load

```bash theme={null}
# Verify model file exists
ls ~/.cache/goose/models/

# Check format (must be .gguf)
file ~/.cache/goose/models/qwen2.5-coder-7b-instruct.gguf

# Re-download if corrupted
goose download-model qwen2.5-coder-7b-instruct --force
```

### Out of Memory (OOM)

```yaml theme={null}
# Reduce context size
context_size: 4096  # Instead of 32768

# Use smaller quantization
# Download Q4_K_M instead of Q5_K_M

# Reduce GPU layers
n_gpu_layers: 24  # Instead of -1
```

### Slow generation

```yaml theme={null}
# Increase batch size
n_batch: 1024  # Instead of 512

# Enable flash attention
flash_attention: true

# Use all CPU cores
n_threads: -1

# Offload more to GPU
n_gpu_layers: -1
```

### Poor quality responses

```yaml theme={null}
# Use higher quantization (Q6 or Q8)
# Increase temperature
temperature: 0.8

# Adjust penalties
repeat_penalty: 1.15
frequency_penalty: 0.1
```

## Advanced: Custom Model Registry

Define custom models in `~/.config/goose/local_models.yaml`:

```yaml theme={null}
models:
  my-custom-model:
    path: /path/to/model.gguf
    context_size: 16384
    temperature: 0.7
    n_gpu_layers: -1
    description: "My fine-tuned model"
```

Use it:

```bash theme={null}
goose configure set GOOSE_MODEL my-custom-model
```

## Implementation Details

### Source Code

* Engine: `crates/goose/src/providers/local_inference/inference_engine.rs`
* Model registry: `crates/goose/src/providers/local_inference/local_model_registry.rs`
* Native tools: `crates/goose/src/providers/local_inference/inference_native_tools.rs`
* Emulated tools: `crates/goose/src/providers/local_inference/inference_emulated_tools.rs`
* Hugging Face models: `crates/goose/src/providers/local_inference/hf_models.rs`

### llama.cpp Integration

Goose uses the `llama-cpp-2` Rust bindings:

```rust theme={null}
use llama_cpp_2::{
    context::LlamaContext,
    model::LlamaModel,
    sampling::LlamaSampler,
};

// Load model
let model = LlamaModel::load_from_file(path, params)?;

// Create context
let ctx = model.new_context(backend, ctx_params)?;

// Generate
let sampler = LlamaSampler::chain_simple(samplers);
let token = sampler.sample(&ctx, -1);
```

## Resources

* llama.cpp: [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)
* GGUF models: [https://huggingface.co/models?library=gguf](https://huggingface.co/models?library=gguf)
* Quantization guide: [https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/README.md](https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/README.md)
* Goose local inference: `crates/goose/src/providers/local_inference/`