Search for a command to run...
A practical walkthrough of clustering repetitive Slack alerts with TF-IDF and DBSCAN, wrapping Python from a Go service, and why classic ML beats an LLM for deduplication.
No comments yet. Be the first to comment.
A practical guide to scaling PostgreSQL: connection pooling, indexing, partitioning, replicas, sharding, and CDC for real-world systems.
Designing an ephemeral one-on-one chat that can jump from two users to two million without changing a line of code.
Winning Honestbee’s 2016 logistics challenge in Saigon with a BFS-based delivery route optimizer
The tool and process we built to move a TB of live data without downtime and without blocking the team
Guiding your software career toward higher‑value work when AI can write most of the code.
Most teams have a Slack channel full of alerts that everyone eventually mutes. When something breaks, the relevant message is buried under hundreds of copies of the same timeout warning and a pod that's been crash-looping for days.
The noise is repetitive. The same handful of errors repeat, with a timestamp and a pod name swapped in. Repetitive data is compressible, and "group the near-duplicates and show one of each" is a clustering problem with well-established solutions.
I built Alligator, a Slack app that reads a channel over a time range and returns a report that collapses 2,000 alerts into the handful of distinct problems behind them. The Slack integration is the presentation layer. This post is mostly about the ML.
Feeding a few thousand log lines to an LLM to summarize the errors works, but it costs money per run, sends production logs to a third party, and returns a different answer each time. The task here is deduplication, and classic ML handles it well: TF-IDF to turn text into vectors, DBSCAN to group the vectors. Both run locally in a couple of seconds, cost nothing, and are deterministic.
Take the text of every message in the channel, turn each one into a vector, then cluster the vectors so near-identical messages group together. Three steps: vectorize, cluster, read off the groups.
Clustering works on numbers, not strings. TF-IDF (Term Frequency–Inverse Document Frequency) converts a set of documents into vectors:
error, then error carries no information about which cluster a message belongs to.The result is that the terms distinguishing one kind of alert from another get the most weight, which is what clustering needs.
The implementation is a few lines with scikit-learn:
from sklearn.feature_extraction.text import TfidfVectorizer
def vectorize_logs(logs):
vectorizer = TfidfVectorizer(max_features=1000)
return vectorizer.fit_transform(logs)
The standard NLP preprocessing pipeline has five steps:
Timeout and timeout count as the same word.timeout, and timeout match.the, is, a.running, runs, and ran collapse to run.TfidfVectorizer performs steps 1–3 by default: it lowercases, strips punctuation, and tokenizes on word boundaries with no extra code. This project relies on those defaults and skips steps 4 and 5.
Stop-word removal targets English prose. Log lines are not prose; they are strings like CrashLoopBackOff, connection refused, and context deadline exceeded. Standard stop-word lists rarely match anything in this data, and the terms that actually saturate it (error, warning, failed) are already down-weighted by IDF. A stop-word pass would be redundant here.
Stemming helps when matching natural language that inflects, such as running versus ran. Log lines repeat near-verbatim and do not inflect, so stemming adds a dependency and risks mangling identifiers (Booting and booted both collapse to boot) with no gain in cluster quality.
The point is to run the full pipeline against the data at hand and keep the steps that improve results. A log clusterer and a sentiment classifier start from the same five steps and keep different ones.
max_features=1000 caps the vocabulary at the 1,000 highest-signal terms. Logs have a long tail of unique tokens, since every UUID and timestamp is new, and without a cap the vectors grow to tens of thousands of low-value dimensions. Capping keeps clustering fast and the vectors focused on terms that recur.
The output is a sparse matrix, one row per message and one column per term. A given alert contains only a handful of the 1,000 terms, so storing the zeros would be wasteful. scikit-learn keeps the matrix sparse end to end, which is part of why this is fast.
One domain-specific step would help more than any textbook one: normalizing log tokens before vectorizing. Right now pod-7f8a9 and pod-3b2c1 are distinct tokens, which pushes otherwise-identical alerts apart. Masking the dynamic parts (timestamps to <TS>, UUIDs to <UUID>, numbers to <NUM>) collapses them into one token and tightens every cluster. It is domain-specific, which is why it is not on the generic list.
With the vectors ready, the next step is grouping them. k-means is the common default and is a poor fit here for two reasons.
First, it requires the number of clusters up front. The number of distinct problems in the channel is the unknown the report is meant to surface. Setting k=5 when there are 12 problems merges unrelated errors.
Second, k-means assigns every point to a cluster. DBSCAN does not. It is density-based: it grows a cluster only where points are packed closely together, and points without enough close neighbors are labeled noise (-1).
For this problem the noise label is useful output, not discarded data. The 2,000 repeated timeouts form a dense cluster that can be collapsed. A single panic: nil pointer dereference that appeared once has no dense neighborhood, so it lands in noise. That singleton is the rare, un-triaged event worth a human's attention. DBSCAN separates the repetitive bulk from the one-offs.
The clustering call:
from sklearn.cluster import DBSCAN
def cluster_logs_to_labels(X, eps=0.5, min_samples=5):
db = DBSCAN(eps=eps, min_samples=min_samples, metric="cosine", n_jobs=-1).fit(X)
return db.labels_
Three parameters carry the weight:
metric="cosine". Euclidean distance is the wrong choice for TF-IDF vectors. A long log line and a short one that mean the same thing sit far apart in Euclidean space because one has more words. Cosine distance measures the angle between vectors, comparing which terms a message emphasizes rather than its length. For text, cosine is usually the right metric.eps=0.5. The distance threshold for two points to count as neighbors. With cosine distance the range is 0 (identical) to 1 (nothing in common). Lower values produce stricter, more numerous clusters; higher values merge everything into one. 0.5 worked on this data: loose enough to catch the same error across different pods, tight enough to keep distinct problems apart.min_samples. How many neighbors a point needs to anchor a cluster. Production runs at 3, so an alert has to fire at least three times to count as a recurring pattern. The benchmark uses 5. This is the first parameter to adjust when tuning.n_jobs=-1 uses all available cores.
The rest is bookkeeping: walk the labels, drop each message into its cluster, and send everything labeled -1 to the noise list.
clusters = {}
noise = []
for label, log in zip(labels, logs):
if label == -1:
noise.append(log)
else:
clusters.setdefault(str(label), []).append(log)
That is the ML core: two scikit-learn calls and a loop. No training run, no GPU, no model to host, no API key. It processes a channel in seconds and returns the same result every time.
The app is written in Go, which suits a small concurrent HTTP service. The ML lives in Python because that is where the mature libraries are. The two connect without a microservice, gRPC, or a second container. The Go service runs the Python script as a subprocess and pipes JSON over stdin and stdout.
cmd := exec.Command(pythonPath, pythonScriptPath)
cmd.Stdin = bytes.NewBuffer(inputJSON)
var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
return nil, fmt.Errorf("error running Python script: %v\nStderr: %s", err, stderr.String())
}
var output ClusterResult
if err := json.Unmarshal(out.Bytes(), &output); err != nil {
return nil, err
}
The Python side reads JSON from stdin, writes JSON to stdout, and sends all logging to stderr so it never corrupts the data channel:
input_data = json.loads(sys.stdin.read())
logs = input_data["logs"]
X = vectorize_logs(logs)
labels = cluster_logs_to_labels(X, input_data.get("eps", 0.5),
input_data.get("min_samples", 5))
# progress and debug info goes to stderr, results go to stdout
print(f"Processing {len(logs)} logs", file=sys.stderr)
json.dump(output, sys.stdout)
The workload is a batch job that runs for a few seconds and exits. A subprocess with a JSON contract is simple to reason about and to test (echo '{...}' | python tfidf_dbscan.py), and it ships both languages in one container, with the Go binary and Python venv side by side. No service mesh, no second deploy.
The project had two goals: solve the alert-noise problem, and understand TF-IDF and DBSCAN well enough to implement them. So I started in Go and wrote both algorithms by hand.
That path taught me the fundamentals. Implementing TF-IDF and DBSCAN from scratch forces you to work through the sparse vectors, the distance metric, and the neighborhood expansion rather than treat them as a library call. It also stalled. Go has limited tooling for vector and matrix computation, so I was building the numeric primitives myself, and the implementation was dense, naive, and single-threaded. Processing about 4MB of logs took nine minutes. The equivalent Python runs in seconds.
At that point I had what I wanted from writing it by hand. The remaining work was optimization that scikit-learn already does: its TF-IDF and DBSCAN are years of tuned, vectorized, sparse-matrix-aware C behind a Python API. Rewriting that in Go was not the goal; solving the problem was. Switching the ML to Python did both, so the project succeeded on both counts.
For numerical and ML work, the library ecosystem often matters more than the language. Python's libraries are fast because the scientific-computing community has spent decades optimizing them.
The Slack integration is thin. The flow:
conversations.history.The respond-now, compute-later split is the part of the Slack integration worth reusing. Slack's 3-second timeout means anything slower than trivial has to run asynchronously:
// respond immediately with a link to the report, which is not ready yet
api.PostEphemeral(channelId, userID, slack.MsgOptionText(
fmt.Sprintf("Processing analytic. Checkout the report at %s/reports/%s",
hostURL, reportKey), false))
// run the clustering in the background
go func() {
output, err := analytic.RunClustering(converted, 0.5, 3)
if err != nil {
slog.Error("Error analyzing texts", "error", err)
return
}
report.RenderReport(reportKey, *output)
}()
The report page is server-rendered with templ. It sorts clusters by size so the loudest problems appear first, shows one sample per cluster with the rest behind a collapsible, and lists the noise separately.
The service runs on Cloud Run, deployed from master. A push triggers Cloud Build, which builds the single Go and Python image, pushes it to Artifact Registry, and Cloud Run rolls it out with zero downtime. Reports live in a Cloud Storage bucket with a 7-day lifecycle so old ones expire on their own.
The highest-value change is log-token normalization: masking timestamps, UUIDs, and numbers before vectorizing to tighten clusters. After that, feeding each cluster rather than the full stream to an LLM for a one-line summary. Cheap, deterministic ML compresses 2,000 lines into a handful of groups, and the model only sees the groups, which keeps the cost low.
Beyond that: larger time windows, better monitoring, and a smoother install for the Slack marketplace.