I Built an Observability Platform in Go Because Datadog is Overkill

Here's the thing about observability tools: the good ones are expensive, and the free ones are basically just fancy log viewers. I've spent way too much time looking longingly at Datadog demos, knowing I could never justify the cost for my projects. When you're working on smaller projects, side businesses, or just want to understand what's happening in your applications, you're stuck between console.log debugging and enterprise pricing.

So I did what any reasonable developer would do: I built my own.

The Problem: Enterprise Tools for Non-Enterprise Budgets

Working at Fischer Identity and previously at Adaptive Computing, I got spoiled by enterprise observability tools. When something broke, I could jump into Datadog, trace the exact request path, correlate logs with metrics, and usually find the issue in minutes. It's powerful stuff.

But then I'd go home and work on personal projects or help friends with their startups, and suddenly I'm back to console.log debugging and hoping for the best. The gap between "nothing" and "enterprise-grade" felt huge.

Logs with context - Not just text dumps, but structured data I could actually query
Performance metrics - Response times, error rates, the basics
Distributed tracing - Because microservices are everywhere now
Self-hosted - No vendor lock-in, no surprise bills
Actually fast - If the monitoring tool is slow, what's the point?

Enter Go-Insight: Observability That Doesn't Break the Bank

Go-Insight started as a weekend project but quickly grew into something I actually wanted to use in production. The core idea was simple: build a lightweight, self-hosted observability platform that gives you 80% of what Datadog offers for 20% of the complexity.

The Tech Stack

I chose Go for the backend because native concurrency makes handling multiple clients trivial, single binary deployment is still magical, and the ecosystem with Gorilla Mux and pgx driver had everything I needed.

PostgreSQL for storage because ACID compliance matters when debugging production issues, strategic indexes turned O(n) queries into O(log n), and JSON support gives flexible metadata storage without NoSQL complexity.

Docker for deployment because of consistency across environments, isolation without dependency conflicts, and easy scalability by adding more instances behind a load balancer.

The Architecture: Simple but Effective

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Your App  │───▶│ Go-Insight  │───▶│ PostgreSQL  │
│             │    │   API       │    │  Database   │
└─────────────┘    └─────────────┘    └─────────────┘
                           │
                           ▼
                   ┌─────────────┐
                   │  Dashboard  │
                   │   (Web UI)  │
                   └─────────────┘

The API exposes simple REST endpoints: POST /logs for ingesting structured log data, POST /metrics for storing performance metrics, POST /traces for creating distributed traces, and GET /logs?service=api&level=ERROR for querying your data.

Performance: The Numbers That Matter

Here's where Go-Insight gets interesting. These aren't theoretical benchmarks—this is real performance data from my test environment:

Sub-10ms response times for filtered log queries (with proper indexing)
100+ concurrent clients handled without breaking a sweat
~5ms average for service-based queries (the most common use case)

The Secret Sauce: Strategic Database Indexing

Most logging platforms slow down as data grows. I solved this with composite indexes that PostgreSQL loves:

-- Service-based queries (90% of real usage)
CREATE INDEX idx_logs_service_timestamp ON logs(service_name, timestamp DESC);

-- Error investigation
CREATE INDEX idx_logs_level_timestamp ON logs(log_level, timestamp DESC);

-- Distributed tracing
CREATE INDEX idx_logs_trace_id ON logs(trace_id) WHERE trace_id IS NOT NULL;

The result? 10-100x performance improvement over naive table scans. A 1 million record query that used to take 45 seconds now returns in 5ms.

Real-World Usage: What It Actually Looks Like

Here's how I integrate Go-Insight into a typical .NET API:

// Simple log ingestion
public async Task LogError(string message, Exception ex)
{
    var logData = new {
        service_name = "user-api",
        log_level = "ERROR",
        message = message,
        metadata = new {
            exception = ex.Message,
            stack_trace = ex.StackTrace,
            user_id = GetCurrentUserId()
        }
    };
    
    await _httpClient.PostAsJsonAsync("http://go-insight:8080/logs", logData);
}

// Performance tracking
public async Task<IActionResult> GetUser(int id)
{
    var stopwatch = Stopwatch.StartNew();
    
    try {
        var user = await _userService.GetUserAsync(id);
        await LogMetric("get-user", 200, stopwatch.ElapsedMilliseconds);
        return Ok(user);
    }
    catch (Exception ex) {
        await LogMetric("get-user", 500, stopwatch.ElapsedMilliseconds);
        await LogError("Failed to get user", ex);
        throw;
    }
}

What I Learned Building This

Simplicity Wins - The temptation was to build every feature Datadog has. Instead, I focused on the 20% of features that solve 80% of problems. Logs, metrics, and basic tracing cover most debugging scenarios.
Performance is a Feature - A slow observability tool is worse than no tool. I spent significant time optimizing queries and building efficient indexes. The sub-10ms response times aren't just nice-to-have, they make the tool actually useful.
Self-Hosting is Liberating - No vendor lock-in means I can modify the platform for my specific needs. Need a custom dashboard? Build it. Want to store additional metadata? Just add a column.
Go is Perfect for This - The combination of great performance, simple deployment, and excellent HTTP handling made Go the obvious choice. The entire platform compiles to a single binary that just works.

The Road Ahead

Go-Insight is still evolving. Current priorities include real-time streaming with WebSockets, advanced alerting based on log patterns, multi-language SDKs (currently building the .NET client), and Grafana integration for advanced visualizations.

Want to Try It?

Go-Insight is open source and production-ready. You can spin up a complete observability stack in under 5 minutes:

git clone https://github.com/NathanSanchezDev/go-insight
cd go-insight
docker-compose up -d

That's it. You now have log aggregation and querying, performance metrics tracking, distributed tracing, a web UI for investigation, and a REST API for integration.

The Bottom Line

You don't need to choose between console.log and enterprise pricing. Go-Insight proves you can have production-grade observability without the enterprise price tag.

Is it as feature-complete as Datadog? No. But for 90% of debugging scenarios, it's more than enough. And when you need something custom, you can just build it yourself.

Sometimes the best tool is the one you control.

@NathanS