> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/block/goose/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry & Observability

> Monitor Goose with OpenTelemetry for traces, metrics, and logs

Goose includes comprehensive observability through OpenTelemetry (OTel), providing traces, metrics, and logs for production monitoring and debugging.

## Overview

Goose's telemetry system provides:

* **Distributed tracing**: Track requests across agents, providers, and extensions
* **Metrics**: Monitor performance, token usage, and error rates
* **Structured logs**: Debug issues with contextual information
* **Flexible exporters**: OTLP, console, or custom backends

## Quick Start

### Enable OpenTelemetry

```bash theme={null}
# Set OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Run Goose
goose session start
```

Telemetry data is automatically sent to your OTLP collector.

### Local Testing with Console Exporter

```bash theme={null}
# Output to stdout for debugging
export OTEL_TRACES_EXPORTER=console
export OTEL_METRICS_EXPORTER=console
export OTEL_LOGS_EXPORTER=console

goose session start
```

## Configuration

### Environment Variables

#### Global Settings

| Variable                   | Default | Description                    |
| -------------------------- | ------- | ------------------------------ |
| `OTEL_SDK_DISABLED`        | `false` | Disable all telemetry          |
| `OTEL_SERVICE_NAME`        | `goose` | Service name in traces         |
| `OTEL_RESOURCE_ATTRIBUTES` | -       | Additional resource attributes |

#### Endpoint Configuration

```bash theme={null}
# All signals
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Per-signal endpoints
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:4318/v1/metrics
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://localhost:4318/v1/logs
```

#### Signal-Specific Settings

```bash theme={null}
# Enable/disable individual signals
export OTEL_TRACES_EXPORTER=otlp    # or "console", "none"
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp

# Timeout
export OTEL_EXPORTER_OTLP_TIMEOUT=10000  # milliseconds

# Metrics temporality (cumulative, delta, lowmemory)
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

# Log level
export OTEL_LOG_LEVEL=info  # trace, debug, info, warn, error
```

### Configuration File

Set in `~/.config/goose/config.yaml`:

```yaml theme={null}
# OTLP endpoint
otel_exporter_otlp_endpoint: http://localhost:4318

# Timeout (milliseconds)
otel_exporter_otlp_timeout: 10000

# Service name
otel_service_name: goose-production

# Resource attributes
otel_resource_attributes: deployment.environment=prod,team=ai
```

<Info>
  Configuration file settings are promoted to environment variables at startup. Environment variables take precedence.
</Info>

## Signal Configuration

### Traces

Traces track request flows through Goose:

```bash theme={null}
# Enable traces
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
```

**Trace structure:**

```
Session Start
  ├─ User Message
  │   ├─ Provider Request (Anthropic/OpenAI)
  │   │   └─ HTTP Call
  │   ├─ Tool Call (developer__list_files)
  │   │   └─ MCP Server Request
  │   └─ Agent Response
  └─ Session End
```

**Trace attributes:**

* `session.id`: Session identifier
* `provider.name`: AI provider (anthropic, openai, etc.)
* `model.name`: Model used
* `tool.name`: Tool invoked
* `extension.name`: MCP extension

### Metrics

Metrics track performance and usage:

```bash theme={null}
# Enable metrics
export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:4318/v1/metrics
```

**Built-in metrics:**

* `goose.session.duration`: Session length (histogram)
* `goose.provider.tokens.input`: Input tokens per request (counter)
* `goose.provider.tokens.output`: Output tokens per request (counter)
* `goose.provider.latency`: Provider response time (histogram)
* `goose.tool.invocations`: Tool call count (counter)
* `goose.tool.duration`: Tool execution time (histogram)
* `goose.errors`: Error count by type (counter)

### Logs

Structured logs with trace correlation:

```bash theme={null}
# Enable logs
export OTEL_LOGS_EXPORTER=otlp
export OTEL_LOG_LEVEL=info
```

**Log levels:**

* `trace`: Very detailed debugging
* `debug`: Debugging information
* `info`: General informational messages
* `warn`: Warning messages
* `error`: Error messages

**Log attributes:**

* `trace_id`: Correlate with traces
* `span_id`: Specific span within trace
* `session_id`: Session identifier
* `module`: Rust module path

## Integration Examples

### Jaeger (Tracing)

```bash theme={null}
# Run Jaeger all-in-one
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

# Configure Goose
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_TRACES_EXPORTER=otlp

# View traces at http://localhost:16686
```

### Prometheus + Grafana (Metrics)

```bash theme={null}
# Run OpenTelemetry Collector
docker run -d --name otel-collector \
  -p 4318:4318 \
  -v $(pwd)/otel-config.yaml:/etc/otel-config.yaml \
  otel/opentelemetry-collector:latest \
  --config=/etc/otel-config.yaml

# Configure Goose
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_METRICS_EXPORTER=otlp
```

**otel-config.yaml:**

```yaml theme={null}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
```

### Elastic Stack (Full Observability)

```yaml theme={null}
# docker-compose.yml
version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  apm-server:
    image: docker.elastic.co/apm/apm-server:8.11.0
    ports:
      - "8200:8200"
    environment:
      - output.elasticsearch.hosts=["elasticsearch:9200"]
```

```bash theme={null}
# Configure Goose
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
```

### Datadog

```bash theme={null}
# Run Datadog Agent
docker run -d --name datadog-agent \
  -e DD_API_KEY=<your-api-key> \
  -e DD_SITE=datadoghq.com \
  -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318 \
  -p 4318:4318 \
  datadog/agent:latest

# Configure Goose
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=goose
export OTEL_RESOURCE_ATTRIBUTES=env:production,version:1.0.0
```

## Implementation Details

### Source Code

* OTel setup: `crates/goose/src/otel/otlp.rs`
* Module: `crates/goose/src/otel/mod.rs`
* Initialization: Called from `goose-cli` and `goose-server` main functions

### Initialization

From `crates/goose/src/otel/otlp.rs`:

```rust theme={null}
pub fn init_otlp_layers(
    config: &crate::config::Config,
) -> Vec<Box<dyn Layer<Registry> + Send + Sync>> {
    let mut layers = Vec::new();

    // Create layers for enabled signals
    if let Ok(layer) = create_otlp_tracing_layer() {
        layers.push(layer.boxed());
    }
    if let Ok(layer) = create_otlp_metrics_layer() {
        layers.push(layer.boxed());
    }
    if let Ok(layer) = create_otlp_logs_layer() {
        layers.push(layer.boxed());
    }

    // Set up trace context propagation
    if !layers.is_empty() {
        global::set_text_map_propagator(TraceContextPropagator::new());
    }

    layers
}
```

### Resource Attributes

Goose automatically includes:

```rust theme={null}
Resource::builder_empty()
    .with_attributes([
        KeyValue::new("service.name", "goose"),
        KeyValue::new("service.version", env!("CARGO_PKG_VERSION")),
        KeyValue::new("service.namespace", "goose"),
    ])
    .with_detector(Box::new(EnvResourceDetector::new()))
    .build()
```

Additional attributes from `OTEL_RESOURCE_ATTRIBUTES`:

```bash theme={null}
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,team=ai,region=us-west"
```

### Signal Detection

Goose determines which signals to enable:

```rust theme={null}
pub fn signal_exporter(signal: &str) -> Option<ExporterType> {
    // 1. Check OTEL_SDK_DISABLED
    if env::var("OTEL_SDK_DISABLED") == Ok("true".to_string()) {
        return None;
    }

    // 2. Check OTEL_{SIGNAL}_EXPORTER
    if let Ok(val) = env::var(&format!("OTEL_{}_EXPORTER", signal.to_uppercase())) {
        return match val.as_str() {
            "otlp" => Some(ExporterType::Otlp),
            "console" => Some(ExporterType::Console),
            _ => None,
        };
    }

    // 3. Check for OTLP endpoint
    if env::var(&format!("OTEL_EXPORTER_OTLP_{}_ENDPOINT", signal.to_uppercase())).is_ok()
        || env::var("OTEL_EXPORTER_OTLP_ENDPOINT").is_ok()
    {
        return Some(ExporterType::Otlp);
    }

    None
}
```

## Filtering

### Trace Filtering

By default, Goose captures:

* All spans at INFO level and above
* DEBUG level for Goose modules

Customize with `RUST_LOG`:

```bash theme={null}
export RUST_LOG=debug,goose::providers=trace
```

### Metrics Filtering

Metrics are captured for:

* INFO level and above
* Events marked with metric fields

### Log Filtering

Log level determined by:

1. `RUST_LOG` (highest priority)
2. `OTEL_LOG_LEVEL`
3. Default: `info`

```bash theme={null}
export OTEL_LOG_LEVEL=debug
```

## Custom Telemetry

Add custom spans and metrics in your extensions:

```rust theme={null}
use tracing::{info_span, instrument};

#[instrument(name = "my_custom_operation", skip(data))]
async fn process_data(data: &str) -> Result<()> {
    let _span = info_span!("processing", data_len = data.len()).entered();
    
    // Your logic
    info!(operation = "process", "Processing started");
    
    Ok(())
}
```

## Disabling Telemetry

### Complete Disable

```bash theme={null}
# Disable all OpenTelemetry
export OTEL_SDK_DISABLED=true

# Or disable individual signals
export OTEL_TRACES_EXPORTER=none
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
```

### Custom Distributions

For custom Goose distributions, disable in code:

```rust theme={null}
// Skip OTel initialization
let layers = vec![]; // Don't call init_otlp_layers
```

## Performance Impact

OpenTelemetry overhead:

* **Console exporter**: \~5-10% CPU
* **OTLP exporter**: \~2-5% CPU + network I/O
* **Memory**: \~10-50 MB for buffering

<Tip>
  For production, use OTLP with batching (default) for minimal overhead.
</Tip>

## Troubleshooting

### No telemetry data

```bash theme={null}
# Verify endpoint is reachable
curl http://localhost:4318/v1/traces -X POST

# Check environment variables
env | grep OTEL

# Enable debug logging
export RUST_LOG=debug,opentelemetry=trace
```

### Spans not correlating

Ensure trace context propagation:

```bash theme={null}
# Goose sets this automatically, but verify
export OTEL_PROPAGATORS=tracecontext
```

### High memory usage

Reduce batch size:

```bash theme={null}
export OTEL_BSP_MAX_QUEUE_SIZE=1024
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256
```

## Resources

* OpenTelemetry Specification: [https://opentelemetry.io/docs/specs/otel/](https://opentelemetry.io/docs/specs/otel/)
* Rust SDK: [https://github.com/open-telemetry/opentelemetry-rust](https://github.com/open-telemetry/opentelemetry-rust)
* Goose implementation: `crates/goose/src/otel/`
* Example integrations: [https://opentelemetry.io/ecosystem/integrations/](https://opentelemetry.io/ecosystem/integrations/)