# Building a data analytic Slack App with Machine Learning

# Clustering Slack alerts with TF-IDF and DBSCAN

Most teams have a Slack channel full of alerts that everyone eventually mutes. When something breaks, the relevant message is buried under hundreds of copies of the same timeout warning and a pod that's been crash-looping for days.

The noise is repetitive. The same handful of errors repeat, with a timestamp and a pod name swapped in. Repetitive data is compressible, and "group the near-duplicates and show one of each" is a clustering problem with well-established solutions.

I built Alligator, a Slack app that reads a channel over a time range and returns a report that collapses 2,000 alerts into the handful of distinct problems behind them. The Slack integration is the presentation layer. This post is mostly about the ML.

## Why not an LLM

Feeding a few thousand log lines to an LLM to summarize the errors works, but it costs money per run, sends production logs to a third party, and returns a different answer each time. The task here is deduplication, and classic ML handles it well: TF-IDF to turn text into vectors, DBSCAN to group the vectors. Both run locally in a couple of seconds, cost nothing, and are deterministic.

## The ML pipeline

Take the text of every message in the channel, turn each one into a vector, then cluster the vectors so near-identical messages group together. Three steps: vectorize, cluster, read off the groups.

### Step 1 — Text into vectors with TF-IDF

Clustering works on numbers, not strings. TF-IDF (Term Frequency–Inverse Document Frequency) converts a set of documents into vectors:

- Term frequency weights words by how often they appear in a given message.
- Inverse document frequency down-weights words that appear in most messages. If every line contains `error`, then `error` carries no information about which cluster a message belongs to.

The result is that the terms distinguishing one kind of alert from another get the most weight, which is what clustering needs.

The implementation is a few lines with scikit-learn:

```python
from sklearn.feature_extraction.text import TfidfVectorizer

def vectorize_logs(logs):
    vectorizer = TfidfVectorizer(max_features=1000)
    return vectorizer.fit_transform(logs)
```

#### Preprocessing: the standard pipeline and what this problem needs

The standard NLP preprocessing pipeline has five steps:

1. Lowercase, so `Timeout` and `timeout` count as the same word.
2. Strip punctuation, so `timeout,` and `timeout` match.
3. Tokenize, splitting each string into individual words.
4. Remove stop words: high-frequency filler like `the`, `is`, `a`.
5. Stem, reducing words to their root so `running`, `runs`, and `ran` collapse to `run`.

`TfidfVectorizer` performs steps 1–3 by default: it lowercases, strips punctuation, and tokenizes on word boundaries with no extra code. This project relies on those defaults and skips steps 4 and 5.

Stop-word removal targets English prose. Log lines are not prose; they are strings like `CrashLoopBackOff`, `connection refused`, and `context deadline exceeded`. Standard stop-word lists rarely match anything in this data, and the terms that actually saturate it (`error`, `warning`, `failed`) are already down-weighted by IDF. A stop-word pass would be redundant here.

Stemming helps when matching natural language that inflects, such as `running` versus `ran`. Log lines repeat near-verbatim and do not inflect, so stemming adds a dependency and risks mangling identifiers (`Booting` and `booted` both collapse to `boot`) with no gain in cluster quality.

The point is to run the full pipeline against the data at hand and keep the steps that improve results. A log clusterer and a sentiment classifier start from the same five steps and keep different ones.

`max_features=1000` caps the vocabulary at the 1,000 highest-signal terms. Logs have a long tail of unique tokens, since every UUID and timestamp is new, and without a cap the vectors grow to tens of thousands of low-value dimensions. Capping keeps clustering fast and the vectors focused on terms that recur.

The output is a sparse matrix, one row per message and one column per term. A given alert contains only a handful of the 1,000 terms, so storing the zeros would be wasteful. scikit-learn keeps the matrix sparse end to end, which is part of why this is fast.

One domain-specific step would help more than any textbook one: normalizing log tokens before vectorizing. Right now `pod-7f8a9` and `pod-3b2c1` are distinct tokens, which pushes otherwise-identical alerts apart. Masking the dynamic parts (timestamps to `<TS>`, UUIDs to `<UUID>`, numbers to `<NUM>`) collapses them into one token and tightens every cluster. It is domain-specific, which is why it is not on the generic list.

### Step 2 — Grouping with DBSCAN

With the vectors ready, the next step is grouping them. k-means is the common default and is a poor fit here for two reasons.

First, it requires the number of clusters up front. The number of distinct problems in the channel is the unknown the report is meant to surface. Setting `k=5` when there are 12 problems merges unrelated errors.

Second, k-means assigns every point to a cluster. DBSCAN does not. It is density-based: it grows a cluster only where points are packed closely together, and points without enough close neighbors are labeled noise (`-1`).

For this problem the noise label is useful output, not discarded data. The 2,000 repeated timeouts form a dense cluster that can be collapsed. A single `panic: nil pointer dereference` that appeared once has no dense neighborhood, so it lands in noise. That singleton is the rare, un-triaged event worth a human's attention. DBSCAN separates the repetitive bulk from the one-offs.

The clustering call:

```python
from sklearn.cluster import DBSCAN

def cluster_logs_to_labels(X, eps=0.5, min_samples=5):
    db = DBSCAN(eps=eps, min_samples=min_samples, metric="cosine", n_jobs=-1).fit(X)
    return db.labels_
```

Three parameters carry the weight:

- `metric="cosine"`. Euclidean distance is the wrong choice for TF-IDF vectors. A long log line and a short one that mean the same thing sit far apart in Euclidean space because one has more words. Cosine distance measures the angle between vectors, comparing which terms a message emphasizes rather than its length. For text, cosine is usually the right metric.
- `eps=0.5`. The distance threshold for two points to count as neighbors. With cosine distance the range is 0 (identical) to 1 (nothing in common). Lower values produce stricter, more numerous clusters; higher values merge everything into one. `0.5` worked on this data: loose enough to catch the same error across different pods, tight enough to keep distinct problems apart.
- `min_samples`. How many neighbors a point needs to anchor a cluster. Production runs at 3, so an alert has to fire at least three times to count as a recurring pattern. The benchmark uses 5. This is the first parameter to adjust when tuning.

`n_jobs=-1` uses all available cores.

The rest is bookkeeping: walk the labels, drop each message into its cluster, and send everything labeled `-1` to the noise list.

```python
clusters = {}
noise = []

for label, log in zip(labels, logs):
    if label == -1:
        noise.append(log)
    else:
        clusters.setdefault(str(label), []).append(log)
```

That is the ML core: two scikit-learn calls and a loop. No training run, no GPU, no model to host, no API key. It processes a channel in seconds and returns the same result every time.

## Wiring Python into a Go service

The app is written in Go, which suits a small concurrent HTTP service. The ML lives in Python because that is where the mature libraries are. The two connect without a microservice, gRPC, or a second container. The Go service runs the Python script as a subprocess and pipes JSON over stdin and stdout.

```go
cmd := exec.Command(pythonPath, pythonScriptPath)
cmd.Stdin = bytes.NewBuffer(inputJSON)

var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr

if err := cmd.Run(); err != nil {
    return nil, fmt.Errorf("error running Python script: %v\nStderr: %s", err, stderr.String())
}

var output ClusterResult
if err := json.Unmarshal(out.Bytes(), &output); err != nil {
    return nil, err
}
```

The Python side reads JSON from stdin, writes JSON to stdout, and sends all logging to stderr so it never corrupts the data channel:

```python
input_data = json.loads(sys.stdin.read())
logs = input_data["logs"]

X = vectorize_logs(logs)
labels = cluster_logs_to_labels(X, input_data.get("eps", 0.5),
                                   input_data.get("min_samples", 5))

# progress and debug info goes to stderr, results go to stdout
print(f"Processing {len(logs)} logs", file=sys.stderr)
json.dump(output, sys.stdout)
```

The workload is a batch job that runs for a few seconds and exits. A subprocess with a JSON contract is simple to reason about and to test (`echo '{...}' | python tfidf_dbscan.py`), and it ships both languages in one container, with the Go binary and Python venv side by side. No service mesh, no second deploy.

### Why the split, and why start in Go

The project had two goals: solve the alert-noise problem, and understand TF-IDF and DBSCAN well enough to implement them. So I started in Go and wrote both algorithms by hand.

That path taught me the fundamentals. Implementing TF-IDF and DBSCAN from scratch forces you to work through the sparse vectors, the distance metric, and the neighborhood expansion rather than treat them as a library call. It also stalled. Go has limited tooling for vector and matrix computation, so I was building the numeric primitives myself, and the implementation was dense, naive, and single-threaded. Processing about 4MB of logs took nine minutes. The equivalent Python runs in seconds.

At that point I had what I wanted from writing it by hand. The remaining work was optimization that scikit-learn already does: its TF-IDF and DBSCAN are years of tuned, vectorized, sparse-matrix-aware C behind a Python API. Rewriting that in Go was not the goal; solving the problem was. Switching the ML to Python did both, so the project succeeded on both counts.

For numerical and ML work, the library ecosystem often matters more than the language. Python's libraries are fast because the scientific-computing community has spent decades optimizing them.

## The Slack app

The Slack integration is thin. The flow:

![Slack app pipeline](https://cdn.hashnode.com/res/hashnode/image/upload/v1722775174421/98da6b53-80b9-4d5b-906b-dd8a6983b905.png)

1. A user triggers Alligator in a channel. The app opens a modal, built with Slack's Block Kit, asking for a start and end time.
2. On submit, the service fetches the channel history for that window via `conversations.history`.
3. Clustering takes a few seconds and Slack expects a response within three, so the service returns a link to a report that is not ready yet and runs the work in a background goroutine.
4. When clustering finishes, the report is rendered to HTML and written to a Cloud Storage bucket. The user opens the link to see the clusters.

The respond-now, compute-later split is the part of the Slack integration worth reusing. Slack's 3-second timeout means anything slower than trivial has to run asynchronously:

```go
// respond immediately with a link to the report, which is not ready yet
api.PostEphemeral(channelId, userID, slack.MsgOptionText(
    fmt.Sprintf("Processing analytic. Checkout the report at %s/reports/%s",
        hostURL, reportKey), false))

// run the clustering in the background
go func() {
    output, err := analytic.RunClustering(converted, 0.5, 3)
    if err != nil {
        slog.Error("Error analyzing texts", "error", err)
        return
    }
    report.RenderReport(reportKey, *output)
}()
```

The report page is server-rendered with [templ](https://templ.guide/). It sorts clusters by size so the loudest problems appear first, shows one sample per cluster with the rest behind a collapsible, and lists the noise separately.

![Clustering result](https://cdn.hashnode.com/res/hashnode/image/upload/v1722775931870/1913aaa6-b303-493c-a774-04884f26fecd.png)

## Deployment

The service runs on Cloud Run, deployed from `master`. A push triggers Cloud Build, which builds the single Go and Python image, pushes it to Artifact Registry, and Cloud Run rolls it out with zero downtime. Reports live in a Cloud Storage bucket with a 7-day lifecycle so old ones expire on their own.

![GCP architecture](https://cdn.hashnode.com/res/hashnode/image/upload/v1722775491055/b56176f6-4221-41d7-8b32-202689a2ca63.png)

## Takeaways

- For deduplication, TF-IDF and DBSCAN beat an LLM on cost, speed, privacy, and reproducibility. The task defines the tool.
- DBSCAN's noise label is useful output for triage. An algorithm that forces every point into a cluster discards the rare one-off, which is often the message worth reading.
- For numerical work, the library ecosystem is a first-order concern. The nine-minutes-to-seconds difference came from scikit-learn's optimized C, not from the choice of Python.
- A subprocess with a JSON contract kept two languages in one container with no orchestration. It is worth replacing only when it causes a concrete problem.

## What's next

The highest-value change is log-token normalization: masking timestamps, UUIDs, and numbers before vectorizing to tighten clusters. After that, feeding each cluster rather than the full stream to an LLM for a one-line summary. Cheap, deterministic ML compresses 2,000 lines into a handful of groups, and the model only sees the groups, which keeps the cost low.

Beyond that: larger time windows, better monitoring, and a smoother install for the Slack marketplace.