I will try to cover this topic as much as possible, based on my current understanding of databases. I welcome all comments on what I'm missing or covered too shallowly. This post is also how I learn: writing it down forces me to understand each practice deeply, so I'll try to explain the why behind everything, not just the what.

I will pick a specific database to narrow down the scope, and I will guide you through a journey of scaling from an MVP to a large scale system. Think of this post as a guided tour rather than a textbook.

What database are we scaling?

There are many kinds of databases, but in general this will be the main database for any application, it could be SQL based or NoSQL based.

We will start with a naive setup: we have a SaaS app, a couple of users and a dozen tables to serve the main functionality, the web app directly opens connections, no pool, a few indexes.

At this point almost any database is capable, so we will pick Postgres, because it is open source, a multi purpose database, and has a lot of extensions in case we go into a niche.

Good enough at this stage:

• Single VM running Postgres

• For monitoring, we rely on the VM's own monitoring: CPU, RAM, Disk usage. If CPU is constantly high, memory is full, or disk is about to run out, we know something is wrong and go investigate.

• Automatic backup: daily or hourly disk snapshot. We should try to restore from the backup at least once to confirm it works.

When do I need a connection pool?

ERROR 53300: Too many connections

I start to have many API calls and each API creates a new connection to the database, and eventually all the connection slots are exhausted. This error may have many root causes:

• Connections don't close properly

• Long running transactions or queries hold the connection

• max_connections param set too low

• Too many connections to the database (obviously)

Why is Postgres so sensitive about this? Because Postgres forks a whole OS process for every connection. Each connection eats several MB of RAM and adds overhead to internal bookkeeping, even when it's idle. So "just raise max_connections" only delay the problem.

When the app and traffic are still small, the app's builtin connection pool like Node.js pg-pool or Go go-pg is usually enough. But once we scale the number of app instances or serverless functions all connecting to Postgres, the database starts to struggle with thousands of short lived connections. This is where we need a dedicated connection pooler like pgBouncer.

So each app instance already has its local pool, then all the app instances connect to one or more pgBouncer instances. Each pgBouncer instance has a predefined, stable pool of connections to Postgres and multiplexes client requests onto them. That way, Postgres spends its valuable time running queries instead of forking and managing processes.

One thing to understand before deploying pgBouncer: the pool mode.

• Session mode: a client keeps its server connection until it disconnects. Safe, but we don't gain much multiplexing.

• Transaction mode: a server connection is only borrowed for the duration of one transaction. This is where the big win is, but it breaks session-level features: SET commands, advisory locks, LISTEN/NOTIFY, and prepared statements (on older pgBouncer versions). We need to check our app and drivers against this list before rolling it out to production.

Introducing pgBouncer is often the first scaling step for Postgres, long before we talk about sharding or more complex solutions.

How do I know I need indexes?

Answer: we need to measure.

Postgres constantly collects statistics about itself and exposes them as regular views we can query with plain SQL. No extra tool, no dashboard to install, we just connect with psql and SELECT from them. The database observes itself, we only need to ask.

The one to start with is pg_stat_statements. It's an extension that tracks every query shape that runs: how many times it was called, total time, average time, rows returned. "Shape" means parameters are normalized, so WHERE id = 42 and WHERE id = 99 are counted as one entry: WHERE id = $1. It ships with Postgres but is off by default (managed Postgres usually preloads it, then we just run CREATE EXTENSION pg_stat_statements;). Once it's on:

SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

Note the ORDER BY. There are two different questions hiding here:

• ORDER BY mean_exec_time answers "which queries are slow?"

• ORDER BY total_exec_time answers "which queries hurt the database the most?" A 5ms query called 50,000 times per minute does far more damage than one slow 2-second query. Chasing total time instead of the slowest query is one of those small habits that separates guessing from measuring. This practice work on other database as well, not just Postgres.

pg_stat_statements has siblings, all queried the same way:

• pg_stat_activity: what is running right now. First thing to check when "the database is stuck": long-running queries, or connections sitting in idle in transaction (bug: transaction opened, never committed).

• pg_stat_user_tables: per-table stats. A big table with a high seq_scan count is being full-scanned repeatedly, that's the index candidate.

• pg_stat_user_indexes: the opposite direction. An index with idx_scan = 0 after weeks in production does nothing except slow down our writes. We should drop it.

Then when a specific query is suspicious, prefix it with EXPLAIN ANALYZE and Postgres shows we the execution plan: sequential scan or index scan, and where the time actually went. So the troubleshooting loop looks like: pg_stat_activity (what's stuck now) -> pg_stat_statements (what costs the most) -> EXPLAIN ANALYZE (why this one is slow) -> fix -> measure again.

That said, indexes are one of the first things we should think about when designing tables. A good index can turn a painful full table scan into a quick lookup, and in practice it fixes most "my query is slow" issues, at least until the data grows big enough. Even when the table is still small, adding the right indexes early saves we from surprises later.

The trade off is always write performance. Every extra index makes inserts and updates a bit more expensive. For unique indexes it's even worse, because the database has to both find a place for the new data and check that the value is not already there.

Depending on the data and the query pattern, sometimes we don't need to index every row. We can use partial indexes with a condition. For example, in an emails table with status = draft | live | editing and category = newsletter | transactional | notification, maybe only non-draft emails are ever queried by users. In that case we can create an index only where status != 'draft'. Reads are faster because the index is smaller, and writes to draft emails stay cheap because they don't touch the index at all. One catch: the planner only uses a partial index when the query's WHERE clause matches the index condition, so WHERE status = 'live' works, but a query without a status filter won't.

If we really cannot avoid adding more indexes, that's usually a hint to also look at other options: improve the queries, archive old data out of the hot table, or later on use partitioning so each index only has to deal with a smaller slice of data.

If we use a managed Postgres, the service will usually surface slow queries and full‑table scans in its monitoring UI, but it won’t go as far as MongoDB Atlas and auto‑suggest exact indexes. In the Postgres world, index advice tends to live in separate tools (pganalyze, Datadog, pgHero...), while the core database only exposes the raw stats (pg_stat_statements, pg_stat_user_tables, pg_stat_user_indexes) and leaves the decision to us.

What is vertical scaling?

That is simply adding more CPU, RAM and Disk to the primary node until the curve between resources and performance doesn't align anymore.

This is why cloud providers make it easy to click and upgrade to a bigger instance. We just go from 4 vCPU / 16 GB RAM to 16 vCPU / 64 GB RAM and suddenly the system is faster. We can buy some time until we need another scaling strategy.

The trade offs:

• The pricing of the bigger instance, but we use this method for a while since hardware is cheap compared to engineering time. Note that we need at least one or two replicas (more on those later), so the price also multiplies.

• The blast radius gets bigger: a larger backup takes longer to restore, a failover moves more traffic at once, and any mistake affects more data.

• At some point, doubling CPU and RAM does not give us the same win anymore. I haven't operated at that ceiling myself, but the consensus is that vertical scaling alone eventually stalls, and we have to combine it with the other tools in this post: better indexes, partitioning, replicas, and in the extreme case, sharding.

What about caching?

Before adding more database machinery, there is a cheaper question: does this read need to hit the database at all?

A cache like Redis (or even an in-process cache) in front of Postgres can absorb a lot of repetitive read traffic: rendered pages, user sessions, computed aggregates, reference data that changes once a day. The common pattern is cache-aside: try the cache, on miss read from Postgres and write back to the cache with a TTL.

The trade‑off is the famous hard problem: cache invalidation. From my experience, caches solve some problems but introduce others.

Once, we hit a serious production incident where data from one tenant was returned to another. All evidence pointed to Redis: the application logs and keys were correct, but the value that came back belonged to a different tenant. After restarting Redis the issue vanished and we were never able to reproduce it. My best guess is a hardware or memory corruption issue, but we never got a definitive root cause. I mention this not to blame Redis, but to show how caching can create subtle, high‑impact failures even when we think we are using it carefully.

Generally, anything with strong consistency requirements (like show the order user just created) should keep reading from the database. When it works, caching is often the cheapest way to buy time: it cuts read load enough so we don't need replicas or a bigger instance yet. But it is another moving part with its own consistency model, so it deserves as much design thinking as the database itself.

Do I need partitioning?

Before jumping to sharding, there is a practice we can apply first: partitioning. In Postgres, partitioning means one "logical" table is actually split into many smaller physical tables behind the scenes, but they all stay on the same server.

Take an example: I have a table events where the data naturally groups by created_at. I can create a partition:

CREATE TABLE events_2026_07 PARTITION OF events
FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');

From the application, I don't care about events_2026_07 or events_2026_08 at all:

INSERT INTO events (user_id, created_at, payload)
VALUES (123, now(), '{ "action": "click" }');

SELECT * FROM events
WHERE created_at > now() - interval '7 days'
  AND user_id = 123;

Postgres will:

• Route the INSERT into the right partition based on created_at.

• For the SELECT, only touch the partitions that overlap "last 7 days" (this is called partition pruning).

So yes, it's still one server, but it can be faster because Postgres skips work:

• It does not scan a huge single index; it works with smaller per-partition indexes.

• VACUUM / ANALYZE runs on smaller chunks.

• When I want to delete old data, I don't run a giant DELETE, I can just: ALTER TABLE events DETACH PARTITION events_2026_01; DROP TABLE events_2026_01;

Typical patterns are:

• Time-based: one partition per day / week / month.

• Tenant-based: one partition per customer or group of customers.

It's all still one Postgres cluster, one primary, one connection string. Partitioning adds a bit of schema complexity, but compared to full sharding across many databases, it's a much smaller step and already helps with performance and data lifecycle.

But partitioning comes with trade offs: we are adding schema and operational complexity. In more detail:

• Partitions need to be created over time. But we should not do this manually, let's use an extension like pg_partman to automate it.

• Migrations and tooling need to be partition-aware.

• We need to pick a good partition key, and Postgres adds one extra rule: the partition key must be part of every primary key and unique constraint on the table.

  In practice, that means we should not partition directly on a high cardinality column like created_at timestamptz and then use (id, created_at) as the primary key. It works, but it's awkward to query and doesn't give us a clean "id is unique" guarantee anymore.

  A more practical pattern is to add a coarser key and partition on that:

  CREATE TABLE events (
    id          bigint,
    user_id     bigint,
    bucket_date date NOT NULL,
    created_at  timestamptz NOT NULL,
    payload     jsonb
  ) PARTITION BY RANGE (bucket_date);
  
  ALTER TABLE events
    ADD PRIMARY KEY (id, bucket_date);
  

  Now bucket_date is the partition key, and (id, bucket_date) is what Postgres uses to enforce global uniqueness across partitions. If the application really wants to treat plain id as the identifier and query with WHERE id = ..., we either add a separate unique/index on id, or accept that "id is globally unique" is enforced by our code, not by the database.

Partitioning becomes interesting when we deal with data on a hot disk (NVMe) and cold disk (SSD, HDD): the latest partition stays on the hot disk for frequent queries, the older partitions move to the cold disk to save cost, but all the data stays in the same database for the occasional query. In practice it looks like this:

1. Mount different disks on the machine, for example:

   1. /mnt/nvme_hot -> fast, expensive

   2. /mnt/ssd_cold -> slower, cheaper

2. Tell Postgres "this folder = this tablespace":

   CREATE TABLESPACE hot_ts LOCATION '/mnt/nvme_hot';
   CREATE TABLESPACE cold_ts LOCATION '/mnt/ssd_cold';
   

3. When creating a partition, put it on the right tablespace:

   CREATE TABLE events_2026_07 PARTITION OF events
   FOR VALUES
   FROM ('2026-07-01') TO ('2026-08-01') TABLESPACE hot_ts;
   

4. Now all the files for events_2026_07 live physically on the NVMe disk. When that partition becomes "cold", move it: ALTER TABLE events_2026_07 SET TABLESPACE cold_ts; Postgres will copy the table's files from /mnt/nvme_hot to /mnt/ssd_cold under the hood.

Two honest caveats on this practice: it only works on self-hosted Postgres, managed services like RDS or Cloud SQL don't let we mount disks or define custom tablespaces. And SET TABLESPACE takes an ACCESS EXCLUSIVE lock for the whole copy, so on a big partition that is real downtime.

That's enough about partitioning. In practice, most teams try everything inside a single Postgres cluster before jumping to sharding. Partitioning is already a noticeable step up in complexity, but it still keeps one primary, one backup story, one connection string.

When do I need replicas?

The first obvious reason to add replicas is to reduce read load on the primary. At some point one node is doing too much: it has to handle every write and also every read (dashboards, exports, background jobs, admin tools). With one or more replicas we can send some of the read traffic away from the primary and keep it focused on writes and the most critical queries.

But replicas are not only about load, they are also a basic high availability (HA) strategy. If the primary dies, requests to that node will fail or stall until something promotes a replica and points traffic there. In practice we use extra tools (Patroni, Stolon, cloud control planes) that run on top of Postgres and use a consensus store like etcd or Consul (which internally run algorithms like Raft) to decide which replica becomes the new primary, and to deal with the old primary when it comes back up. This is a sophisticated problem and it isn't worth solving ourselves when the industry already has the standard, or we simply use managed Postgres from a cloud provider.

Of course, replicas come with trade offs. Replication is asynchronous by default, so there is always some lag. If a user writes data and then we immediately read from a replica, they might not see their change yet. So we need to be careful which queries are allowed to hit replicas

• Strong consistency paths (like "show me the order I just created") should read from the primary

• Things that can be a few seconds behind (reports, analytics pages, background jobs) can go to replicas. This is the "read your own writes" problem.

This is where the OLTP / OLAP split appears. OLTP is the normal product traffic: small, focused reads and writes on fresh data. OLAP is the heavy stuff: big scans and aggregations. If we run OLAP queries directly on the primary, they will compete with OLTP and affect the performance. A common pattern is: primary for OLTP, one small replica ready for failover, and maybe one "analytics" replica that serve the requests from dashboards or batch jobs. Even then, replicas still share the same schema and indexes as the primary, so if the analytics workload grows too much we usually push it into a separate OLAP store via CDC.

When do I consider sharding?

If the system is multi-tenant or the data can be grouped by some clear key, we can start thinking of sharding. Sharding basically means instead of one big Postgres cluster, we split the data across multiple databases, usually by a shard key like tenant_id or region.

A simple example: each tenant (or group of tenants) goes to its own database. All the tables look the same, but tenant A lives on shard 1, tenant B lives on shard 2, and so on. From the app side, we either have a small routing layer, or the app itself decides which connection string to use based on the shard key. The main win is isolation: a noisy tenant or a huge customer only overloads its own shard, not everybody else. Storage and write load are also spread across machines instead of hammering a single primary.

One warning on picking the shard key: it should spread the writes. Sharding by time (one shard per day or month) sounds tempting but it's an anti-pattern for write scaling, all new writes hammer the newest shard while the old shards sit idle. Shard by tenant or a hash of some ID, and partition by time within each shard if needed.

Of course, the trade offs are much bigger than partitioning. Now we don't just manage one Postgres cluster, we manage many: more backups, more replicas, more monitoring, more things to upgrade. Queries that need data from multiple shards become more annoying: either we avoid them, or we have to fan out and aggregate in the app or some separate job. Moving tenants between shards and keeping them balanced is also extra work. We trade operational risk for performance.

Because of that, we should only start to seriously consider it when:

• A single well-tuned Postgres cluster (with connection pooling, good indexes, partitioning, and vertical scaling) is still not enough for the write/load pattern, or

• We have a couple of big tenants that clearly deserve their own hardware so they don't hurt everyone else.

When we reach that stage, we should be ready to live with the operational cost of many Postgres clusters.

CDC & ETL to OLAP / other stores

Postgres records every change in its write-ahead log (WAL) before applying it. Physical replication (the replicas above) works by shipping this WAL to standby servers. Change data capture (CDC) reads the same WAL but through a different mechanism, logical decoding, which turns the low-level changes back into a stream of row-level insert/update/delete events. Once people had that stream, they found many uses for it: event driven architecture, the outbox pattern, notifications, and feeding OLAP stores.

We just subscribe to the change log. Tools like Debezium or cloud-native services read the WAL from Postgres and turn it into an event stream of inserts/updates/deletes.

Once we have that stream, we can build an ETL pipeline:

• Extract: listen to the change events coming from Postgres (for example via Debezium into Kafka).

• Transform: clean up or enrich the events in a stream processor or simple workers (cast types, denormalise some fields, join with reference data).

• Load: write the transformed data into a store that is better for analytics: a columnar warehouse, a time-series database, a search index, etc.

The OLTP database stays focused on small, transactional queries. The OLAP side (warehouse, time-series, whatever we pick) gets a copy of the data that is usually a few seconds behind, but is shaped and indexed for heavy reads.

CDC isn't free, the trade offs:

• We add more moving parts. Instead of "app → Postgres → dashboards", now we have app → Postgres → WAL → CDC connector → message bus (Kafka, Kinesis) → ETL jobs → warehouse / OLAP store → dashboard. Every box can break, lag, or need upgrades. Operating this pipeline is a real ongoing cost.

• One subtle risk with CDC is how it uses replication slots. A logical consumer (like Debezium) tells Postgres "keep WAL from this position onward for me", and Postgres will not discard any WAL segments that this slot has not acknowledged yet. If the consumer falls behind badly or is quietly shut down, that slot never advances, so every new write generates more WAL that cannot be cleaned up. Over time those files pile up on disk. In the worst case they fill the volume and the primary cannot write new WAL or commit transactions, effectively taking it down. The practical fix is operational: monitor replication slot lag as a first‑class metric (for example, pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) from pg_replication_slots), alert when a slot stops moving, and drop abandoned slots so Postgres can safely recycle old WAL.

• Our system turns from consistent to eventually consistent.

• Schema changes become harder to manage: one change in the Postgres schema needs updates through the whole pipeline.

• We increase the cost of every write: replicating data, writing logs, computation in the ETL pipeline.

• We face the general problems of distributed systems: handling duplicates, retries, backfills, idempotent workflow design, message ordering, reprocessing events after fixing a bug in the ETL (the event sourcing problem).

What about operational concerns?

All of the scaling practices above only work if we can see what is going on in the database and change things safely. The basic concerns are:

Observability

We need at least charts, logs, and alerts around some key metrics: connections, IOPS, CPU, RAM, disk usage, replication lag, and slow queries. With a bare metal Postgres setup, the loop is: running Postgres -> exporter process -> expose HTTP endpoint -> Prometheus periodically calls that endpoint -> stores time series data -> shows charts. Then we set up alert rules like:

• CPU > 80% for 5 minutes, fire alert.

• Free disk < 20%, fire alert.

• Replication lag > 30 seconds for 2 minutes, fire alert...

Managed Postgres does the same thing under the hood, it just hides the exporter + Prometheus bits and gives you "CPU graph + alarms" as a service.

By sending alerts to Slack or PagerDuty, we can detect issues early and start thinking about the scaling practices.

Observability brings up capacity planning. Instead of waiting for "database is down", we watch leading indicators: CPU trend, connection count, table and index size, replication lag, storage usage. When the curves start to bend, we plan changes (new indexes, bigger boxes, replicas, partitioning) before we hit the wall.

Autovacuum and bloat

This deserves its own mention because it's one of the most common ways a real Postgres install falls over at scale.

Postgres uses MVCC (Multi-Version Concurrency Control): an UPDATE or DELETE doesn't remove the old row, it marks it dead and leaves it in place. The dead rows (bloat) are cleaned up later by autovacuum. On a quiet database the defaults are fine. On a write-heavy table, autovacuum can fall behind: tables and indexes grow with dead space, scans get slower, and everything looks like a mystery performance problem even though the indexes are "right".

Two things to do:

• Watch bloat and autovacuum activity (pg_stat_user_tables shows dead tuples and last vacuum times). Tune autovacuum to be more aggressive on hot tables, the default settings are conservative.

• Know that transaction ID wraparound exists: if vacuum is prevented from running long enough (for example by a forgotten long-running transaction or an abandoned replication slot), Postgres will eventually force a shutdown to protect the data. It's rare, but we should monitor for it: watch n_dead_tup in pg_stat_user_tables, and never let transactions or replication slots sit idle for days.

Safe schema changes

A schema change on production can require long locks and downtime, we can't just simply ALTER TABLE. We need a better way: add columns in a compatible way -> backfill in small batches -> only drop old columns when nothing reads them anymore.

We also need to propagate the change through the ETL pipeline if any, which means it isn't just a technical change: we need to align the schema change with other teams, make sure all of them follow the new schema, then drop the old one. This sounds obvious, but in a big company, the communication issue usually leads to more problems than the technical one.

Backup and restore tests

You may have heard the stories of big companies that did backups frequently but never tried to restore, and when they had to, it didn't work and became a disaster. Recently I learned about 2 numbers that indicate a good backup strategy: RPO and RTO.

• RPO (Recovery Point Objective): How much data can we afford to lose? For example 15 minutes, then we need a base backup and WAL archives to have point in time recovery.

  • Point-in-time recovery is about being able to restore your database to an exact moment in the past, not just to when a backup file was taken. In Postgres terms, it works like this in practice:

    • You take a base backup (a full snapshot of the database at some time, e.g. 02:00).

    • You also keep all WAL segments that record every change after that backup.

    • If something bad happens at 10:37 (buggy deploy, DELETE FROM users, disk crash), you:

    • Restore the base backup from 02:00,

    • Then replay WAL forward until just before the bad thing happened, for example 10:36:55. That's called Point-In-Time Recovery (PITR): "give me the database as it looked at 2026-07-08 10:36:55".

• RTO (Recovery Time Objective): How long can we afford to be down while restoring the data? For example, if our RTO is 30 minutes, but a full restore from S3 to a fresh VM already takes 45 minutes, then we need to review our backup plan. One way to know is to actually do a restore from time to time:

  • Take one of our real backups.

  • Spin up a new Postgres instance in a separate environment.

  • Restore the backup and replay WAL to some point in time.

  • Measure how long it takes until the database is ready to accept connections.

That test tells us the real RTO number. If the number is too big, we either change the backup strategy (more frequent base backups, faster storage) or we accept that our RTO is worse than we wanted. For small side projects a daily snapshot is probably fine. For serious systems we almost always need continuous archiving and regular restore drills.

The iterative mindset

Scaling is not a one time project. We measure, notice the potential problem, make the change, measure again and keep going.

Answering a scaling problem requires understanding the problem first. Let me play out how this looks in an Q&A style conversation, because that back-and-forth of questions is the method.

Question: We need to write a billion data points and adding an index will slow the writes. We need to keep the writes scalable, what should we do?

Before answering, I need to question back. What is the time range for that billion data points, per day, per hour, per minute? How big is a record and what is in it? What is the current schema and the existing indexes? And why do we need the new index?

Q: Say it's a billion per day.

That's around 12K writes per second on average. A well-tuned single Postgres with vertical scaling and batched inserts can likely handle it. No exotic architecture needed yet.

Q: What if it's closer to a billion per hour?

Then the next question is: do we need ACID on this data, or can we accept eventual consistency? If we can accept it, put a high throughput message queue like Kafka in front to buffer the spikes, and insert into the database gradually through the queue. If we truly need transactional consistency at that rate, we shard, and the shard key must spread the writes: shard by tenant or a hash of some ID, never by time. Time-based sharding means every new write hammers the newest shard while the others sit idle. Partition by time within each shard instead.

Q: And the index we wanted to add?

What does it's for, OLTP or OLAP? If it's a non-negotiable OLTP index, consider a partial index to reduce the number of rows it covers, and archive or partition old, cold data so the index only covers the current working set. If it's for OLAP queries, we don't optimize heavy aggregations on the OLTP tables, we can use CDC / ETL to push the data into a warehouse or OLAP-friendly store and index it there.

Q: Does the shape of the data change the answer?

Yes. If the data is mostly unstructured documents, we can store it as JSONB in Postgres, but that should be treated as a convenience feature, not a full replacement for a document database. For heavy document workloads, purpose-built engines like MongoDB or Couchbase are usually a better fit.

If the data is structured, ask how it is queried and how many fields are repetitive versus unique per row. A high ratio of repeating values plus heavy analytical queries points to a columnar or warehouse-style store such as ClickHouse, Amazon Redshift, or Google BigQuery, which can compress and scan those columns much more efficiently. Postgres is a great fit for structured, transactional data where we care about ACID; for large unstructured or purely analytical workloads, a document or columnar database will usually be the better primary home.

I built a small project to take both of those seriously — an ephemeral one-on-one chat. A host opens a room, a guest joins, the two talk in real time, and the moment either leaves, the room and every message in it are deleted. No history, no archive. The feature is small on purpose. The architecture is not, and that's the point of this post: the same codebase that serves two users is designed to serve millions, and the path between those two numbers is adding boxes, not rewriting code. I'm confident this design carries a million concurrent chats. Here's the reasoning.

The naive design dies twice

Here's the version almost everyone writes first:

1. Save the message to the database.

2. Push the message to the connected clients.

It works every time you test it, and it dies for two independent reasons.

It dies for correctness. The gap between step 1 and step 2 is a crash-shaped hole.

The dual-write problem: when one operation updates two systems that don't share a transaction (a database and a message broker), any crash between the writes leaves them inconsistent, with no clean way to recover.

Crash after the commit but before the push, and the message is saved but never delivered. Push first instead, and a crash leaves it delivered but never saved — gone on the next page load. No ordering of two independent writes survives a crash in the middle.

It also dies for scale, and this is the one I actually care about. In the naive design the request thread does the fanout — it holds the WebSocket connections and pushes to each recipient inline. That couples two things that must never be coupled at scale:

• Request latency is now proportional to recipient count. Sending into a busy room, or broadcasting the lobby's room list to every idle user, makes the sender's request slow. Fanout size should never be in the critical path of a write.

• The server is now stateful. It holds connections in memory, so you can't just add a second box — connections for a conversation might live on a different instance than the write. You're one node, or you're building sticky sessions and a custom cross-node fanout. You've painted yourself into a single-server corner on day one.

You can't patch your way out of either of these. You design your way out.

One write, then get out of the way

The fix for both problems is the same move: the request does exactly one thing — a single database transaction — and then returns. It never touches the broker, never holds a connection, never fans out.

That's the transactional outbox pattern (well documented by Chris Richardson and the microservices community). In one transaction you write two facts to the database: the business row, and a row in an outbox table describing what to publish and to whom. One transaction, one source of truth — either both land or neither does.

func (o *messageUsecase) Create(ctx context.Context, msg string, publicRoomId, publicUserId uuid.UUID) (model.Message, error) {
	var createdMsg model.Message
	errCreate := o.txManager.UseTx(ctx, func(ctx context.Context) (err error) {
		// ... resolve room and user ...

		createdMsg, err = o.messageRepo.Create(ctx, newMsg)   // (1) the message
		if err != nil {
			return
		}

		// ... work out who needs to receive it ...

		// (2) the delivery intent — same transaction, no broker call
		err = o.outboxUsecase.PublishMessageToUsersInRoom(ctx, createdMsg, user, usersToPublish)
		return
	})
	return createdMsg, errCreate
}

Look at what this buys at scale. The request does bounded, local work — render the message for each recipient (O(N) CPU, but cheap and never more than the people in a room) and a single batched insert (O(1) round-trips). It holds no WebSocket connections and performs no broadcast fanout itself, so the app tier stays stateless and scales sideways. The expensive, unbounded part — actually reaching everyone — happens asynchronously downstream of Kafka, where it can be absorbed and parallelized. And the one path where fanout really is large — broadcasting the lobby's room list — is a single outbox row addressed to many channels, so even that stays O(1) on the write.

And because the request never holds a connection, the app tier is completely stateless. Every instance does nothing but write to Postgres and return. That single property is what lets you put fifty app servers behind a load balancer with no coordination between them. Stateless is the precondition for horizontal scale, and the outbox is what makes the app stateless.

Why Kafka — the part that actually scales to millions

Once the app just records intent and walks away, something has to turn those outbox rows into delivered messages. That something is a change-data-capture pipeline, and the choice of Kafka in the middle is the deliberate scale decision of the whole system.

Every box on that diagram scales on its own axis, and Kafka is why. Specifically:

• Throughput through partitioning. Kafka is a distributed, partitioned, replicated log. You scale write and read throughput by adding partitions and brokers — there's a well-trodden path from thousands to millions of messages per second. The delivery tier's ceiling is Kafka's ceiling, and Kafka's ceiling is very far away.

• A durable buffer, so spikes don't cascade. When a traffic spike hits, or Centrifugo briefly falls behind, messages queue durably in the log instead of backing up into the app or getting dropped. Kafka turns a spike into a slightly longer log, not an outage. That backpressure boundary is exactly what keeps a real-time system standing during the moments that matter.

• Parallel consumers via consumer groups. Centrifugo reads as a consumer group; partitions distribute across its instances. More connected users → more Centrifugo nodes → more partitions consumed in parallel. The WebSocket tier scales independently of the app tier and the database.

• Per-room ordering for free. Every outbox row is stamped with its room id as the partition key:

  // partitionId = message.RoomId
  o.outboxRepo.CreatePublishOutbox(ctx, channel, renderedMessage, message.RoomId)
  

  That's the seam that lets a single room's messages stay strictly ordered on one partition while millions of other rooms spread across the cluster for throughput. Ordering and parallelism usually pull against each other; partition-by-room is how you get both.

• New consumers cost the write path nothing. The log is retained, so the day you want moderation, search indexing, analytics, or an audit trail, you attach another consumer group and replay. At a million chats you will want those — and none of them require touching the code that sends a message.

This is also why I use CDC instead of polling the outbox table. Polling works at small scale, but at large scale it puts a relentless query load on the database and floors your latency at the poll interval. Debezium reads Postgres' write-ahead log — the database already wrote it, so capture is nearly free and nearly instant. It's the version of "drain the outbox" that doesn't fight you as the numbers grow.

Ephemerality is a scaling feature, not just a product one

The obvious objection to the diagram above: Postgres is your single stateful core — isn't it the bottleneck? It's the right question, and the design has two answers.

First, rooms are independent. Two conversations share nothing, which makes the room id a natural shard key. Past the ceiling of one primary, you partition or shard by room and the write load spreads cleanly, because nothing needs to join across rooms.

Second — and this is the part I like most — the data is ephemeral by design, and ephemerality bounds the working set. When a conversation ends, the room and its messages are deleted. The database holds only active chats, never the full history of everything ever said. A product with a million concurrent chats stores a million chats' worth of rows, not ten years of accumulated archive. The one stateful tier stays small precisely because nothing is allowed to persist. The feature that makes the product feel private is the same property that keeps the core scalable.

HTML over the wire — and why I didn't reach for React

The default for an app like this in 2026 is a React SPA talking to a JSON API. It's a fine architecture for some products. It also drags in a specific set of bugs that have nothing to do with your feature and everything to do with the shape of the design — and I've debugged all of them more than once:

• Two renderers, one truth. The server owns the data; the client owns the markup. So display logic — formatting, "show this badge when…", "hide the delete button unless you're the host" — gets written twice, in two languages, and drifts. The two halves of your app slowly start to disagree.

• The state-sync problem. This is the big one. The client keeps a copy of server state and has to keep it fresh: cache invalidation, stale reads, optimistic updates that need rolling back when the server says no. You've created a distributed-state problem inside your own UI. Half of what React-Query and Redux do exists to manage a problem you only have because you chose to duplicate the state.

• Contract drift. Rename a field on the server and a cached SPA bundle in someone's browser breaks. Now you're versioning APIs and writing contract tests to police an agreement between two things you wrote yourself.

• Permissions in the wrong place. "Hide the button unless you're the host" is a client check — but the client can't be trusted, so the real rule has to live on the server too. Put it only on the client and it's a security hole; put it in both and it drifts.

• Round-trip waterfalls. A screen needs data from three endpoints, so it's three sequential fetches and three spinners — or a bespoke per-screen endpoint, or GraphQL and its own complexity tax.

Every one of these is a consequence of splitting the view across the wire as data. So I didn't split it. The server is the single source of truth and it sends rendered HTML — the hypermedia-driven approach, HTML over the wire, revived by htmx and Carson Gross's Hypermedia Systems, DHH's Hotwire (literally HTML Over The WIRE), and Phoenix LiveView. The browser's only job is to drop the HTML into the page. There's no client copy of state to sync, no JSON contract to drift, and permissions are decided in the one place that also renders the markup.

I wanted to prove it holds all the way through a real-time system, so even the outbox payload carries HTML, not a message object:

renderedMessage, err := o.roomPresenter.RenderMessage(ctx, message, sender, receiver)  // -> HTML string
_, err = o.outboxRepo.CreatePublishOutbox(ctx, channel, renderedMessage, message.RoomId)

The thing travelling Debezium → Kafka → Centrifugo → the socket is a <div>, not a {}. And the same ChatMessage templ component renders in all three places a message can appear — the initial page load, the sender's HTTP response, and the live socket push:

// the sender's own message, rendered straight into the HTTP response
err = messageComponent.ChatMessage(newMsg, user, user).Render(ctx, w)

One definition of what a message looks like, three delivery mechanisms, zero contract to keep in sync. It composes with the scale story, too: the delivery tier stays dumb — Kafka and Centrifugo shovel an opaque payload with no schema or logic in the hot path — while the one smart step, rendering, happens on the stateless app tier you can add boxes to.

The trade-off

None of this is free, and it's the wrong choice for some apps:

• You ship more bytes per update — HTML is heavier than JSON for the same data (though gzip narrows it, and you're not also shipping a megabyte of framework).

• The server does the rendering, spending CPU the client would otherwise spend. Cheap and cacheable here, but real.

• It's a poor fit for genuinely client-heavy UIs — offline-first, local computation, canvas-style interactivity (Figma, Maps, a spreadsheet). Those need client state; hypermedia would fight them.

• There's no free JSON API for, say, a future native mobile client. If I ever need one, I add it then — I'm not paying for it now.

I took those costs on purpose, because this app sits exactly where hypermedia wins: it's server-state-driven — the UI is a reflection of rooms and messages that live in the database, not a client-side computation — and it's built by a person. Trading flexibility I don't need for the elimination of an entire second codebase, a state-sync layer, and whole categories of drift bugs is not a close call. The best code is the code I didn't have to write to keep two copies of the truth agreeing.

The seams that keep each tier swappable

A system with this many collaborators — database, transaction, broker, renderer, object store — only stays sane if the dependencies point inward and are expressed as interfaces. The use cases depend on RoomRepo, TxManager, not on pgx or Centrifugo. That's what let me unit-test the transactional logic with hand-rolled fakes and no database at all, and it's what lets any one tier be replaced as the scale story evolves.

My favourite small seam is how a transaction crosses layers without leaking its type everywhere. The TxManager stashes the live transaction in the context; every repository quietly asks "is one in flight? use it; otherwise use the pool":

func GetTxIfAny(ctx context.Context, p *pgxpool.Pool) Querier {
	tx, ok := ctx.Value(helper.TxKey).(pgx.Tx)
	if ok {
		return tx
	}
	return p
}

A repository doesn't know whether it's inside UseTx. The use case owns the transactional boundary; the repositories just honour it. Ten lines now, instead of threading a tx argument through every signature later.

What's still the frontier

Confidence isn't pretending the work is finished — it's knowing exactly where the next problems live and seeing that the architecture already has room for them:

• Delivery is at-least-once, not exactly-once. A consumer can replay after a crash, so the client needs to de-duplicate by message id. That's a known, bounded piece of work — and at-least-once is the correct default; it's the guarantee that never loses a message.

• Postgres sharding is real work past a point. The shard key (room id) and the bounded, ephemeral working set are already in place; turning the crank is an infrastructure step, not a re-architecture.

• Identity is a cookie, not real auth. Fine for a demo, first on the list for anything real.

None of these change the shape of the system. They're the next boxes, not the next rewrite — which is the whole claim.

The takeaway

Anyone can wire up a WebSocket for two users. The design question is whether those two users and two million share a codebase or a tombstone. This one shares a codebase, because every decision was made against the larger number: the request does one transaction and gets out of the way, so the app tier is stateless and scales sideways; Kafka absorbs the fanout, buffers the spikes, orders each room, and feeds consumers you haven't written yet; ephemerality keeps the one stateful core small; and the smart rendering happens once, far from the tier that has to reach everyone.

The WebSocket was the easy afternoon. An architecture that grows by adding boxes instead of starting over — that's the month. And that month is the work that's still ours.

Public GitHub repo for the code

https://github.com/finnng/ephemeral-chat

Some screenshots

• The main screen, every users will see the lobby to select the room or create a chat room to wait for the other to join

• The other user will see the room in the lobby immediately.

• Then they can click to the room name to join and start the chat. The room now will disappear from the lobby. Only two of them can chat

  

• The view from the host

  

It was fun building this project. I'm thinking of hosting it somewhere to let people come to chat just for fun ;)

As I remember, the problem was about finding the fastest route to deliver goods, considering a few factors such as the distance between addresses, the weight of the goods, and the value of the goods. We assigned a score in several dimensions, from high to low:

• Distance: nearer deliveries get higher scores

• Weight: lighter packages get higher scores

• Value: higher-value goods get higher scores

I don’t remember the exact formula, but we computed a score for each data point to decide which location to visit next in the route.

For the algorithm, we used a simple BFS-based approach on top of this scoring. It wasn't a perfect solution to the full vehicle routing problem, but under hackathon time pressure, a clear heuristic plus a straightforward search was good enough to find fast routes and keep us competitive on the leaderboard.

This was in 2016, when AI was nowhere to be found in our toolbox. We coded everything by hand, with no tests, and had to make it runnable within a few hours and submit it to an API that powered a real-time leaderboard. It felt like a game to us, watching the leaderboard update with our names and our rivals’. In the end, the top 10 submissions were all ours. We successfully delivered all the goods using the fastest routes.

We won the first prize. The challenge was fun, and my teammate and I had a great bonding experience. We weren’t the smartest individuals, but together we achieved something much bigger than what either of us could have done alone.

I learned that day that when you create a good environment where people feel challenged but also safe and have fun doing their work, they will perform at their best.

Here was the certificate

On the leaderboard, the higher at the bottom was better.

vhit was the name of our school, We named the team vhit to honor our school of Vietnam - Korea University of Information and Communication Technology that we all graduated in the same year of 2012.

I only had this blurry picture of the band, the guy on the left is Huỳnh Minh Tuấn, the guy on the right is Phùng Đông Hưng.

• Realtime by default: RethinkDB gave you native changefeeds, so apps could subscribe to live query updates instead of building polling or oplog-tail logic around MongoDB.

• Better fit for live apps: It was designed for dashboards, collaborative tools, multiplayer systems, and other “realtime web” use cases where MongoDB was more of a general document store.

• ReQL was more elegant: RethinkDB’s query language was chainable and embedded in your host language, so it felt cleaner than assembling MongoDB-style query documents and driver calls.

• More powerful querying: RethinkDB emphasized joins, subqueries, and aggregations in the database, which made complex document queries easier than MongoDB’s then-limited relational story.

• Simpler clustering: Sharding and replication were presented as easy to set up, with a web admin UI that reduced operational friction.

• Stronger consistency story: RethinkDB marketed itself as “a MongoDB done right,” stressing MVCC, non-blocking writes, durability by default, and better correctness trade-offs.

• More attractive developer experience: Reviews from the time repeatedly called out ease of use, clear docs, and “just works” realtime behavior as reasons people got excited about it.

• MongoDB needed extra plumbing: If you wanted similar realtime behavior in MongoDB, you usually had to add another layer yourself, which is why RethinkDB felt more complete for that niche.

Internally, the product team also aimed to build a real-time web app. We chose React to build the SPA frontend. And as the team said, the backend also using a real-time database “makes perfect sense".

Then on a rainy day out of the blue, RethinkDB company announced that they will shut down https://rethinkdb.com/blog/rethinkdb-shutdown/. No more new features or bug fixes. They left us with one of the biggest technical debts. Migrating away from it is obvious, but one of the company's most competent features is built on top of it. The company is also a startup, so releasing new features to get to the top of the market is the most important thing. And the plan to migrate from it was delayed for years, until the day the product is mature, and the coming of AI makes it feasible.

In this post I’ll walk through why none of the obvious migration approaches fell apart under our constraints, how we designed a Data API to hide the migration from the rest of the team, and how a purpose-built Go migrator with distributed locks and time-windowed checkpoints moved terabytes of live data with only minutes of downtime.

Problem statements

Migrate away from RethinkDB to MongoDB is one of our multi-year plans to modernize the system. With over TB of data, 200+ tables, and the 99.9% downtime requirement, we need a solid migration process and tool that can tolerate errors and run the migration simultaneously on several tables, while maintaining data consistency.

RethinkDB is no longer maintained, so the modern approach like using the CDC doesn’t exist. The nearest thing to CDC is the native RethinkDB change feeds, which is unreliable because the change feeds only work if the consumer is live. If the network blips or the migrator is restarted, then we lose the data changes.

As we migrate data, we still need to serve the production traffic, an event-driven system serving 30M+ events per day, we can only put the system to maintenance mode for like 10 minutes during the night.

Some of the proposed solutions are:

• Dump the data from RethinkDB then import to MongoDB: This doesn't work because the dump requires a long downtime, and the dumping tools for RethinkDB are also slow and unmaintained, the MongoDB import tool is also slow on hundreds of GBs of data, and if there are failures we need to start from the beginning.

• Stream the data directly: this also doesn't work because it can't guarantee the data consistency on live system. Stream data is a point-in-time cursor scan, any document inserted, updated, or deleted after the cursor has already passed it will be silently missed.

• Use RethinkDB change feeds: As I mention above, it is unreliable and the event doesn't guarantee to delivery at least once, if the connection drops, all events that occurred during the gap are permanently lost with no built-in offset or replay mechanism to resume from.

The 10+ years code base is also a challenge when the RethinkDB queries embedded into both the test and code files across 3 languages Nodejs, Go, and C# within million lines of code.

And most importantly, we are not only migrating data in a live production system, we also migrate the code where the rest of the team is still pushing the product feature every day. We can't just block the deployment until we finish the code; we need to merge the code gradually and give the team the room to hot fix their product feature issue if any.

I will break down the problems and solutions, considering the trade-offs involved. Every problem has sub-problems, and so do the solutions. We have made several important decisions, and I hope to have enough time and space to present them all. However, if anything is unclear, feel free to leave a comment. I will reply to it or update the post to address it.

The general solution

The key requirement is how to migrate the data while the team still work on the same code base, build feature that requires database query while the migration code and data change keep moving spontaneously

We considered two approaches

1. Approach A: Direct query replacement behind a migrationStatus flag

// Run a database operation depending on current migration status
procedure runQuery(oldDbOperation, newDbOperation):

  if migrationStatus is NOT_STARTED:
    // operate only on old database
    result = execute oldDbOperation
    return result

  else if migrationStatus is MIGRATING:
    // operate on both databases, in parallel
    resultFromOld = execute oldDbOperation
    execute newDbOperation
    wait until both operations are finished
    // use old database result as the source of truth during migration
    return resultFromOld

  else if migrationStatus is MIGRATED:
    // operate only on new database
    result = execute newDbOperation
    return result

  else:
    // invalid state
    signal error "Invalid migration status"

1. Approach B: Data API over HTTP, like the Supabase API, where all database access goes through a shared HTTP service instead of a direct SDK. So instead of a query like SELECT name FROM users WHERE userId = x, callers just call await readDocumentById('users', id, ['name'])

In the scope of this post, I will only introduce the Data API pattern but not dig deep into it, it is a sophisticated service and worth a separate post to cover the details. What is relevant here is that each table has a migrationStatus field in the migrator's tracking document, and the Data API reads it on every request to decide which database to talk to. During Migrating, every write goes to both databases and reads come from RethinkDB. Once an operator marks a table Migrated, the Data API drops RethinkDB entirely for that table and routes all traffic to MongoDB — with no deployment and no code change required.

We chose the Data API approach after considered several trade offs

• Accessibility over the performance. Other team also need to access the same database, if we choose (1), the other team need to write the new query and the migration themselves. The Data API will expose as the HTTP server so other team can just call the Data API without knowing which database serving the request.

  • We also need the Data API to auto-scale based on workload. Ideally, we will keep it inside a Kubernetes cluster, but Cloud Run offers the same horizontal and vertical scaling ability and the ease of access to other team members. Then, we choose Cloud Run for deployment.

• Fast execution over performance. If we use the migrator in approach (1), we need to replicate this across 3 implementations in Node.js, C#, and Go, not to mention the direct RethinkDB query in the test files across 3 languages. By replacing it all by HTTP calls with the unified interface, we hide all the migration behind. The team can continue to work on the feature without knowing the migration behind the Data API call. If we just replace old queries by the new queries, it costs us the same effort and slows down the execution as the team still works on the same codebase; conflict is inevitable.

The system is mature, so we need to design the solution based on the existing infrastructure and design rules:

• The services and workers running inside the Kubernetes cluster for auto-scaling.

• The workers can't access the database directly; only the service can call the database to prevent the exhausted connection pool by having too many worker instances.

• The database is outside of the Kubernetes cluster to provide easy access to other teams as we shared the database. Ideally, each team will own their database, but in practice, to save costs, we usually use the same database instance.

The database migration tool

Here is the main part that keeps the migration process safe and fast. But first, let's review the requirements again so I can explain the design decisions to satisfy them.

• TB of data is live on production while all the migrations run in the background. We can only tolerate a few minutes down at midnight.

• Data integrity is non-negotiable; rollback is hard because the live data keeps writing to the database.

• Avoid the operational risk of human mistakes.

• The timeline is tight; we need to migrate tables in parallel to speed it up.

• We need some data manipulation before inserting data from RethinkDB to MongoDB because the two systems handle absent values differently. RethinkDB stores explicit null fields inside documents, while MongoDB applications typically treat a missing field and null as distinct things. Keeping those null fields would silently change query behavior, so the migrator recursively strips all null values from every document — including nested objects and arrays — before writing to MongoDB.

• The process should be resilient, tolerant of network issues, server crashes, and running inside Kubernetes to reuse the existing infrastructure and vertically auto-scale.

I came up with the design, in general.

The migration tool is a single Go service: a web dashboard to control migrations and a background worker engine. One binary, deployed in Kubernetes.

Why Go

The migrator needs to run up to at least 20 concurrent workers, sustain steady memory pressure for days at a time, and stay maintainable by a single engineer who may not touch it for months. Node.js was ruled out early — it's hard to hold a long-running process steady under high heap load, and its event loop model makes true parallelism awkward when you need goroutine-per-worker isolation with clean, explicit cancellation. Go's goroutines and sync primitives gave us that naturally, and the context package made cancellation and timeout propagation explicit all the way down the call stack — critical when a crashing pod must release a distributed lock cleanly.

Static typing also mattered. A migration tool is not product code; no one is going to refactor it frequently, and the person who picks it up next might not have full context. Go's compiler catches whole classes of mistakes before they reach production, and Go's compatibility guarantee means the binary compiled today will still build and run on a future toolchain without surprises — no npm audit roulette, no breaking major version upgrades to chase.

Building this tool also sharpened my Go fundamentals in ways I hadn't expected: designing worker pools that are free of race conditions, using sync, context, and log/slog idiomatically instead of pulling in frameworks, and keeping the dependency count low so the tool can outlast the migration itself without rotting.

The dashboard was a deliberate choice too. Instead of a React frontend, a separate API, and a build pipeline to maintain, I used Go's html/template package and wired interactions with htmx. Every action — start, stop, reset, reclaim a lock — is a single HTML attribute and one server endpoint. No JavaScript, no bundler, no frontend dependencies to audit. Combined with Go's embed package for static files and the table config CSV, the entire service — migrator engine, scheduler, and dashboard — ships as one binary. Deploying an update in Kubernetes is rolling out one image.

The table list is an embedded CSV in the binary — table names, collection names, connection strings. On startup the service seeds a tracking document per table into MongoDB. This is the source of truth: migration phase, progress, and lock state for each table. It also computes a safe batch size per table from RethinkDB stats (100MB ÷ avgDocSize) to keep memory usage bounded.

Scheduling and distributed lock

Every 30 seconds, BootstrapWorker loads all Migrating tables, sorts by who ran least recently, and fills up to 20 worker slots. When it wants to start a worker, it does one atomic FindOneAndUpdate in MongoDB to claim the lock. If another pod holds a valid lock, it skips that table. Locks have a 3-minute TTL with a 1-minute heartbeat — if a pod crashes, the lock expires and another pod picks it up.

BootstrapWorker (every 30s)
  ├── worker already running?  ──YES──► skip
  ├── 20 slots full?           ──YES──► wait next tick
  ├── lock acquired?           ──NO───► skip (other pod)
  └──► Start Worker

Workers run one cycle then exit, freeing their slot. The 30-second tick is the loop — this is what lets the scheduler fairly rotate across all 150 tables instead of 20 workers running forever and starving the rest.

What one worker cycle does

The write hook in step 1 is the backbone of the whole sync: RethinkDB automatically stamps __lastModifiedAt on every insert or update, which is what lets the data sync query only what changed since the last cycle.

Each worker runs three steps in sequence. First, it verifies the __lastModifiedAt index exists on the RethinkDB table and creates it if missing. Then it processes any pending deletions that the Data API has already recorded in MongoDB. Finally, it runs a Between range scan on the __lastModifiedAt index for the bounded time window, reads documents in batches, strips null fields from each one, and bulk-upserts them into MongoDB using ReplaceOne with upsert: true, keyed on the table's RethinkDB primary key. The lastBackfillAt checkpoint is written to the tracking document only after the full time window commits successfully — so a crash means at most one 15-minute window is re-processed on the next cycle, not the entire table.

For data sync, the time window advances each cycle with a small backward overlap to handle clock skew and data replication lag, and a 15-minute forward cap so a large backlog gets drained incrementally rather than in one huge scan.

──────────────────────────────────► time
[   previous cycle   ][─5min][  this cycle ≤15min ]
                             ▲                     ▲
                          fromDate                toDate
                                  (lastBackfillAt moves here)

Here is a few of the most asked questions about this tool

Q: Old documents in RethinkDB never had __lastModifiedAt — how does the first sync pick them up?

A: The index creation defaults missing __lastModifiedAt to now at index build time, grouping all old docs under the same timestamp. The first run also has no cap — it scans from the beginning of time to now, so everything gets swept in one pass.

Q: Who writes the deletion tracking records — the migrator?

A: No. The Node.js Data API writes a tracking record into MongoDB before and after every delete from RethinkDB. The migrator only reads that collection. It never observes RethinkDB deletes directly.

Q: What if the migrator crashes mid-table?

A: The lock expires in ≤3 minutes. The next BootstrapWorker tick on any surviving pod claims it and resumes from lastBackfillAt — the last committed checkpoint. At most 15 minutes of work is repeated due to the cycle cap.

Q: What if a document is updated in RethinkDB while the migrator is currently processing that same window?

A: The write hook updates its __lastModifiedAt to now, pushing it past toDate. The next cycle's window picks it up. The 5-minute backward overlap also acts as a safety buffer.

Q: How do you know when a table is fully migrated and safe to cut over?

A: The dashboard shows migratedDocCount vs totalRethinkDocs. When they converge and deletions are drained, the table is a candidate for cutover. The actual switch happens at midnight: we put the system in maintenance mode for a few minutes, deploy the feature flag values that tell the Data API to read from MongoDB for the target tables, and during those minutes the data is identical in both databases so there is no risk window. Then we click Complete in the dashboard to stop the migration workers for those tables and bring production back up. After that we run regression and smoke tests on prod to confirm. We also have a rollback plan if anything looks wrong, but that deserves its own post.

Q: 20 parallel workers still means 20 simultaneous reads on RethinkDB — is that fine?

A: Each worker reads at most ~100MB per batch, not the full table. The 20-slot cap was tuned to the cluster's comfortable read concurrency. It's light, sustained pressure rather than a spike. And we can always increase this limit.

Q: How do you verify the migrated data is identical between RethinkDB and MongoDB?

A: We built a separate validation worker. After every successful read through the Data API, the service asynchronously publishes a verification message to a Pub/Sub topic — fire-and-forget, so it never adds latency to the actual request. A 10-minute deduplication window prevents the same read from flooding the queue.

The worker picks up the message, replays the identical request against both the RethinkDB and MongoDB endpoints concurrently, then normalizes both responses: stripping database metadata fields (_id, __lastModifiedAt), removing nulls, and sorting keys alphabetically for a stable comparison. Then it runs a deep equality check. On any mismatch it generates a detailed diff and logs it.

This only runs on tables still in the Migrating phase, so real production traffic continuously validates the migration without any overhead on already-completed tables.

Results

So far, 100+ tables and over a TB of data have been fully migrated without incident. That doesn't mean the path was smooth — we hit OOM panics in the early weeks and found some bugs along the way. Both were addressed: OOM by tightening the batch size calculation, bugs by patching the worker logic. The service has 95% test coverage across unit and integration tests, and we ran it on staging for months before touching production, so most of the hard surprises were already behind us by the time real data was involved.

Throughput isn't our bottleneck — the migration runs in parallel with the team rewriting direct RethinkDB queries to Data API calls, so we don't rush it. That said, the tool can move roughly a TB within a single working day when given the bandwidth. And because the pull-and-lock model is stateless, scaling is just adding pods — no coordination required.

Observability is deliberately simple: per-table progress on the dashboard, worker errors logged directly into the tracking document, and Kubernetes resource metrics for the rest. We haven't wired up external alerting yet — it wasn't worth the integration cost for an internal tool at this stage — and the dashboard has been enough to catch issues in practice.

Final thoughts

One thing that isn’t obvious from the architecture: before this tool existed, migrating a table meant running bash scripts manually — on a dev machine, a VM, or a Cloud Run instance — each one adjusted by hand per table. Miss a step or run them out of order and the table ends up in an unknown state with no easy recovery. With 200+ tables in the pipeline, that was a real operational risk, not a hypothetical.

The tool replaced all of that with a few clicks on a dashboard. What used to take hours or a full day for a single table now takes minutes. And the company stopped carrying a category of risk it didn’t even know it had.

In the end, this migration wasn’t about some clever trick. It was about surviving a decade of history sitting in an abandoned database, while a 30M+ events/day system kept running and the rest of the team kept shipping. Almost every “simple” idea fell apart once we accounted for downtime limits, consistency, and human error. What worked was a combination of boring, reliable pieces: a Data API in front of the databases, a purpose-built Go migrator with locks and checkpoints, small bounded windows, and a process designed so that failure just meant “retry later,” not “panic at midnight.”

We’re still migrating. But months in, the tool has held up, the team keeps shipping without noticing, and we haven’t had a data incident. For a years-overdue migration running in the background of a live system, that’s good enough.

Strengthen the tech stack and domain expertise

"AI can write all the code, why bother choosing the language?" You may have heard this already, but the answer is simple: a language is tied to its ecosystem, it takes years to become an expert in a tech stack, and we rely on that expertise to judge the solutions provided by AI.

The code looks fine, and all the tests pass, but is it safe to deploy? Not quite. AI makes several architecture decisions no one is aware of, like using long expiration days for the auth token in a fintech product, choosing a low-maintenance npm package from some unknown developer, or heavily investing time in edge cases that never exist in real life.

Taste and judgement

So, being fundamentally strong in your company's tech stack is even more valuable in the AI age. AI can write code, but we still need to review and sometimes rewrite it, and redirect AI toward better code. We are opinionated here, not just accepting any code. Our taste is proven and solid, let it dictate the way new code is written. You're familiar with the system you own, your team's coding conventions, your battle scars. Even though all conventions are written into some instructions, AI can still go off the rails sometimes.

Regarding domain knowledge, AI can't replace anyone outright yet. Even in a scenario of a many-years complex project system, even though we can implement new features or fix bugs tangled in layers of complexity, anyone on the team can just drop in Opus 4.6 and a few lines of requirements and AI can implement a lot of that, it's still not the end of your expertise. Companies still need people to drive development. The metrics below show exactly where that human edge lives.

Drive the output via TDD

This is real-life experience from the team I'm working in, and several others have experienced the same: nothing drives AI output better than the TDD approach. You can also think of spec-driven development in the same mindset, we must understand what we need to do first and force the AI to actually understand what it's going to build.

When you're directing thousands of lines of code generation, you need a forcing function that makes you actually understand what's being built. Tests are that forcing function. -- Martin Fowler

We don't even need to write the tests line by line. Given the existing tests in the system, we can instruct AI to build new tests based on existing patterns and feed it our requirements for the expected output. It needs to write the test first and run it, the test should fail, then write the code and run the test again, keeping fixing the code in a loop until all green. With AI, TDD is easier than ever.

Context Engineering

AI providers keep improving models to handle larger and larger context windows, but that doesn't mean we can put the whole codebase into one prompt. Models don't know where to focus their attention when the provided context is too large. At enterprises, codebases can be millions of lines of code, not to mention that logs and real production data can be large in some cases.

Also, code is not everything; logs, metrics, ticket systems, and human discussions are inputs as well. We need to carefully compose good context for a task: that can mean referencing the relevant ticket, including meeting notes on the solution, querying related data and logs, and embedding them into the prompt. This process is often called context engineering; in an agentic setup, it becomes one part of a broader “harness engineering” discipline that also covers tools, repositories, and feedback loops.

Observability and Debugging

I mapped my debugging skills into an agent and shared it with the team. In general, it's the skill of looking for related data in logs, metrics, and traces. In detail, it includes grabbing traceable information like traceId and entityId, finding the related logs to query in our Elasticsearch database, then locating the relevant code that produces those logs, then asking for a real piece of data, looking for related tickets and discussions around the issue. That is my expertise, collected over years of working on the system, that's not something AI can easily establish without a human to orchestrate it. Not to mention production access: we can't give it to AI directly. Who knows what a hallucinating AI agent might do.

As a result, this AI agent alone reduced my MTTR by 12x, from 1 hour to 5 minutes. In most cases, within 5 minutes I can identify what is wrong with the incident we're facing, compared to hours of investigation before the AI age.

But if I stop feeding my skills to AI, stop providing the correct data through my authority, and stop judging the AI's investigation results, the output becomes meaningless. As I keep improving, the AI also improves through my discoveries.

Metrics

Technical expertise sets the foundation, but metrics are how we keep score. Here's where the human edge becomes measurable.

Coding metrics

Thanks to AI, we can now produce more options and more solutions to the same problem. For code, there are some simple metrics we can measure right after solutions are provided, like Cognitive Complexity.

Cognitive Complexity specifically measures how difficult code is for humans to understand, heavily penalizing nested control structures like if statements, loops, and switches.

We can reinforce it through instructions, but sometimes I still need to edit the code manually to maintain readability. Other metrics we can use are Cyclomatic Complexity, Maximum Nesting Level, and Duplicated Lines (%).

Product metrics

As we, developers, step into higher-value work, the product-centric metrics are what we need to be aware of and contribute to after the coding step. Metrics like feature adoption, clicks, retention, conversion, and search matching rate are usually the KPIs of the product team, but understanding them will bring developers closer to the customer. It's what companies usually dream of: a product engineer.

A product engineer (or product-minded engineer) blends software engineering with product thinking. They code features while considering user needs, business impact, and data, beyond pure tech tasks.

Service Level Metrics

Service Level Metrics are the most important for developers. They're tied into the contract we signed with customers, so they must be satisfied, that's a direct impact on the company. Example:

Take this example from the internet to understand it better:

During a traffic spike: SLI drops to 99.7% (outages from nested if bugs in your API). SLO breached → investigate, error budget exhausted → prioritize fixes over features. SLA holds (still >99.9%), avoiding penalties. Ties back to code reviews: Low Cognitive Complexity prevents such failures.

Metrics lead all the way back to decision-making when prompting, reviewing, and refining code to ensure business value. We haven't given AI this level of thinking yet, and AI also doesn't go to a client and sign the SLA contract, doesn't take charge of the penalties. Only humans are accountable for it. It leaves us with questions like:

• Should we implement this feature at all? What is the benefit?

• Among 3 proposed solutions, what technical debt can we tolerate?

• The solution is clear, but it requires 48 hours of work for a low-value ticket, is it worth it?

• Our biggest client can produce 30M events at peak, what do we need to do to ensure the SLA?

Ownership and Accountability Mindset

AI can't be assigned to PagerDuty and join an incident call at 3:00 AM. It can help in the incident investigation process, but do customers want to show up on a call with an AI agent during an incident? Definitely not. When SLAs are breached and penalties land, that weight falls on a human, and always will.

Based on the foundation of technical expertise and metrics-centric development above, we can see that the more AI absorbs the doing, the more valuable the person who takes responsibility becomes.

Communication and Collaboration

One of my favorite communication skills is translating the technical focused discussions into language that product people can understand, while keeping it relevant and precise. People love the summarize button, but who can ensure that summary is still correct? Only the person sitting between these layers can take responsibility for it.

At this point, I think we can all agree that we can't just prompt all the way to production, and on the other side, engineers can't only work on code when AI can do that part most efficiently. Pure software engineers are elevating to product engineers, who can orchestrate the technical components to satisfy requirements from the product and the customer, and can also communicate well between stakeholders.

I remember the scene from The Matrix Reloaded where Neo tells the Architect that the machines need human beings to survive. Even though AI has assimilated most human knowledge, without us, it would stagnate, waiting for new human discoveries. We are not becoming obsolete. We are becoming the ones who decide what gets built, why it matters, and who answers for it.

That observation shapes every architectural decision I make on a new project. The shape of my system should follow the shape of my team, not the other way around.

Most teams over-engineer day one and pay for it for years. I've worked on projects that turned into maintenance nightmares — some eventually rescued, but only after burning serious time and money. The technical debt followed them long after. It doesn't have to go that way.

When I'm building something new with a small team, I focus on functionality, not scale. A vertically scaled, stateless server behind a good proxy like Traefik gets me much further than I once thought possible. Here's the architecture I reach for today.

My Majestic Monolith

Five principles I treat as non-negotiable:

1. Clean architecture from day one. I keep my business logic independent of my infrastructure. It's just separation of concerns, but the payoff compounds. When I eventually need to extract a service — when team structure or workload shape demands it — only the infrastructure layer changes. The core logic runs the same whether it lives in a VM, a Kubernetes pod, a Lambda, or a Cloud Function.

2. One capable server. I pick a runtime that can fully saturate the hardware I'm paying for and run uninterrupted for months. For a long-lived monolith, I look for efficient use of multiple cores under sustained load, predictable memory behavior over weeks of uptime, and a mature concurrency model. Different languages meet that bar in different ways — I choose one my team can operate confidently for years, not just ship the first feature with.

3. A managed cloud database. Postgres or MongoDB cover almost every use case I've encountered. I let the cloud provider handle scaling, and I sidestep the "shared database vs. database-per-service" debate entirely, because I don't have services yet.

4. A stateless server. Single-server doesn't mean single-instance. When a hot spot emerges — say, the user domain — I spin up more instances of the same codebase, point the load balancer at them, and route that domain's traffic to those instances. I've effectively got a dedicated service without rewriting anything. Token-based auth means any request can land on any instance, so I skip sticky sessions entirely.

5. A push-based message bus. Async work doesn't require a fleet of pods polling a queue. I keep the business logic inside the core, expose it as an API, and configure pub/sub to push messages to that endpoint. Long-running handlers process the work and acknowledge the message.

This gives me message ordering, natural backpressure (the broker slows down when my service is under load), and minimal operational overhead. If a workload is genuinely heavy, I run it on a separate instance of the same monolith dedicated to async tasks.

Why Most Microservices Are Unnecessary

Across multiple projects, I've seen the same pattern: most microservices don't earn their keep — usually because they were created without an organizational reason to exist. They were a copy of someone else's architecture, not a reflection of the team that owns them.

An in-process function call is orders of magnitude faster than an HTTP call, and it avoids network latency, retries, and the long tail of distributed-system failures. Building those failure modes into the system before they're justified is the microservices premium I'm not willing to pay early.

In my experience, code is rarely the bottleneck — the database is. Most API work is data manipulation, waiting on a query, or waiting on a third-party service. Time I'd spend scaling microservices is usually better spent scaling the database.

When a Microservice Actually Earns Its Keep

Two things justify carving out a service:

Resource isolation. Some workloads will starve the rest of the system of CPU or memory if they share a process:

• Email composition, HTML rendering, PDF generation
• Image and video processing
• AI model inference

Team boundaries. Once a domain is owned by a separate team with its own release cadence, decisions, and on-call rotation, Conway's Law catches up. Forcing two teams to coordinate inside one codebase creates more friction than running two codebases. That's the moment to split — not before.

Scaling the Organization, Not Just the System

When I'm leading multiple teams, I still don't start with microservices. I split along domain boundaries first — Conway's Law applied deliberately. Once a team owns a domain, the architecture inside it is their decision; I only care about the contract they expose.

The monolith doesn't survive forever, and that's fine. It survives as long as the team structure justifies it. When the org grows, the system grows with it — but along the seams the team has already drawn, not arbitrary lines I imagined on day one.

My goal is delivering value with the least effort possible, with an architecture that matches the team I have — not the team I'm imagining I'll have someday.

The database

Wakapi provide several choices of database.

Supported databases

Wakapi uses GORM as an ORM. As a consequence, a set of different relational databases is supported.

• SQLite (default, easy setup)

• MySQL (recommended, because most extensively tested)

• MariaDB (open-source MySQL alternative)

• Postgres (open-source as well)

• CockroachDB (cloud-native, distributed, Postgres-compatible API)

• Microsoft SQL Server (Microsoft SQL Server)

I was looking for a free service to host my database and discovered that the Supabase free tier is sufficient for this small application. They offer a managed Postgres service, which I really appreciate as I love using Postgres.

Get started with:

• Unlimited API requests

• 50,000 monthly active users

• 500 MB database space

  Shared CPU • 500 MB RAM

• 5 GB bandwidth

• 1 GB file storage

• Community support

Supabase is a company, and their need for profit is important. I don't encourage spamming their free tier; we only use it here as an experiment. Also, remember to back up your data because we don’t know when it might be stop offering the free package.

To get started with the free package, follow these steps:

1. Visit https://supabase.com/pricing and register for an account.

2. Create a project on the platform.

3. On the home page, click the "Connect" button to obtain the connection string.

Make sure to copy the connection string for the Session mode, as it supports concurrency and is suitable for your app. The Transaction mode is only appropriate for one-time commands.

The Wakapi app on Google Cloud

This is the hard part. You need a credit card to continue 😆

However, if you can afford it, it's really cheap.

Cloud Run is incredibly affordable for low workloads. I created a new Google Cloud account to experiment with Kubernetes, but in October I only used a single Cloud Run instance to host my Wakapi. The total cost for that was 2,733 VND, which is approximately $0.11. That's why I say it feels almost free. Please ignore the free trial credit in the image below, as it expires in three months. However, the monthly fee for Cloud Run remains minimal.

How to set up your app

Steps:

1. Pull the original Docker image.

2. Push it to your Google Cloud Artifact Registry.

3. Create a Cloud Run instance that uses the most recent image you just pushed.

4. Set up the necessary configurations.

Pull the docker image

 docker pull --platform linux/amd64 ghcr.io/muety/wakapi:latest

Remember to pull this arch --platform linux/amd64 If you are on MacOS Apple silicon chip, it will pull the arm64 image and you can’t push it to Artifact register

Push it to your Google Cloud Artifact Registry

After you pull it, your image will be wakapi:latest Now push it to you google project

docker push gcr.io/YOUR_GOOGLE_PROJECT_ID/wakapi:latest

Create a Cloud Run instance that uses the most recent image you just pushed.

Create a new Cloud Run Service deploy from container

Give the required setting

• Select the image you just pushed

• App entry point

• 2GiB RAM and 2 CPU is more than enough

Set up the necessary configurations.

Don’t click Deploy yet, remember your database secret? Wakapi provide all kind of configuration here https://github.com/muety/wakapi?tab=readme-ov-file#-configuration-options

You can set it however you want. Here is mine.

Example:

# Database Configuration
export WAKAPI_DB_AUTOMIGRATE_FAIL_SILENTLY=true
export WAKAPI_DB_TYPE=postgres
export WAKAPI_DB_HOST=aws-0-ap-southeast-1.redacted.supabase.co
export WAKAPI_DB_PORT=5432
export WAKAPI_DB_USER=postgres.redacted
export WAKAPI_DB_NAME=postgres
export WAKAPI_DB_PASSWORD=********  # Replace with your actual database password

# Application Settings
export WAKAPI_LISTEN_IPV4=0.0.0.0
export WAKAPI_ALLOW_SIGNUP=false
export WAKAPI_INSECURE_COOKIES=true

# Security Settings
export WAKAPI_PASSWORD_SALT=********  # Replace with your secure random salt
export WAKAPI_SENTRY_DSN=********    # Replace with your Sentry DSN if using error tracking

# Environment Type
export ENVIRONMENT=production

The most crucial configuration is the database secret. Once you're satisfied with your settings, click "Deploy" and wait a few minutes for your Cloud Run service to boot up. At the top of the page, you will see a URL; this is your public URL to access your Wakapi. You also have the option to purchase a domain and point it to your Cloud Run service. Googling Cloud Run domain mapping for the further instruction.

Setting Up the Client

Wakapi uses the same client as Wakatime, which means that all Wakatime plugins can send data to Wakapi. You have the option to disable data sending to Wakatime and send data exclusively to Wakapi, and that is completely acceptable.

To get started, navigate to your deployed Cloud Run URL and register your first account. Make sure to retrieve the API key and update it in the appropriate location in your system, located at ~/.wakatime.cfg

That pretty much, if you run into any issue, let me know in the comment.

I realized that I was freely providing my valuable coding statistics to them; as a free service, I became the product. I’ve been searching for alternatives and found Wakapi. Wakapi is an open-source project that sends data to your self-hosted server using the same plugins and protocols as Wakatime. It’s quite an ingenious solution. Additionally, since the Wakatime plugin is open-source, we can modify it to send data anywhere we want, and that's perfectly legal.

I set up my server and used it for a while, and I found it fascinating. I'm thinking of contributing to it. Noticed several small problems with this alternative, such as performance issues and a lack of features. So, I found a good first issue from the list to start with:

https://github.com/muety/wakapi/issues

I'm not familiar with the codebase yet, so working on an issue that requires me to walk through it would be a good starting point, rather than just updating the README. The code is written in Go, and Go developers tend to utilize the standard library as much as possible.

In line with the preferences of the repository owner, I think I can do some refactoring, and I found a suitable ticket for that. The issue isn't difficult, but it does involve a lot of changes. Specifically, it requires replacing the logbuch library with slog. Structured logging was introduced in Go version 1.21, making many structured logging libraries obsolete. Therefore, replacing third-party libraries like logbuch is an obvious choice.

I picked this issue: Migrate from logbuch to slog #480

Collaborating with the repository owner fascinates me as well. After a few back-and-forth PR updates, the pull request was merged. Yay 🎉🎉

Follow up by a few more discussion and additional pull request, the issue is resolve.

Contributing to this issue alone makes me the 9th/50 most significant contributor to the project, possibly due to the LOC changes 😜

Thanks to Ferdinand Mütsch, who is such a nice guy. I still use Wakapi every day and have found many improvements I can implement. I just need to find the time.

My take away to start contribute to OSS

• Journey

  Let’s dive into your project. Identify the challenges you face and seek open-source solutions. Utilize these solutions and pinpoint areas for improvement. If you’re familiar with the tech stack, consider reviewing the issue list or submitting an issue ticket to start collaborating with others.

• Expertise

  To begin contributing, you need a certain level of expertise. The open-source software (OSS) you develop will be used by many people, potentially thousands or more, running the code you write on their own infrastructure or computers. We don’t want to create issues for them with any new bugs.

• Be Proactive

  The time frame for a pull request (PR) should be within a few weeks; otherwise, we risk losing context. While it's great that we are using our free time to create open-source software, maintaining a professional approach is what everyone expects in a collaborative environment. Be proactive and assist the maintainer when needed.

That's all for today. I hope you find inspiration to start your journey as an open source contributor. Here is my GitHub profile; let's connect for more interesting projects: https://github.com/finnng

I've always wanted to build a chat system, just for the joy of it. The original plan was simple: set up an old HP Elitedesk as a server, NAT the ports, point a domain to it, and share it with friends. But as I looked at today's job market, I realized this could be more than just a fun project.

These days, job requirements read like a technical encyclopedia. Companies want developers who can:

• Master frontend technologies from service workers to real-time vanilla JS

• Code proficiently in multiple languages (typically Go and JS) with at least 7 years of production experience

• Be an expert in both SQL and NoSQL databases, plus columnar databases like Druid or Cassandra for analytics

• Handle pub/sub systems like Kafka for microservices

• Implement solutions like Debezium or Postgres listen/notify for replication lag

• Set up comprehensive monitoring with logging, tracing, and metrics

• Deploy and manage Kubernetes clusters (CKA certification preferred)

• Build and maintain CI/CD pipelines

• And of course, demonstrate experience with systems handling 100M+ requests per day

And that's just to get past the CV screening - we haven't even gotten to the LeetCode challenges yet!

So, I decided to turn my chat system project into a learning journey. I'm setting an ambitious goal: build a system that can handle 100 million requests per day. Not just because it's a common job requirement, but because it's an excellent way to learn these technologies in a practical context.

In this series, I'll document my journey building a scalable chat application from the ground up. We'll cover everything from frontend implementation to deployment and scaling strategies. No shortcuts, no oversimplification - just real, hands-on experience with the tools and techniques that modern tech companies use.

Let's start with where users first interact with our system: the frontend.

The Unexpected Journey Back

Back in 2012, when I started my career, Node.js was everywhere. The job market was flooded with Node.js opportunities, and I dove straight in. For the next decade, that's where I lived. In doing so, I completely missed the era where people built websites with PHP and jQuery - a gap that would later prove interesting in my HTMX journey.

Beyond Pet Projects: Building Real Systems

Everything changed when I started building a complete system rather than just another small app. Let me tell you - building a system is an entirely different beast from creating "pet" projects where performance, design, and scaling aren't critical concerns. It requires a bird's-eye view while still demanding attention to every line of code.

The scale of the project made me realize something crucial: I needed to minimize the technology stack I had to maintain. Fewer moving parts mean a more stable system. While our backend was solid (even with complex pieces like Kafka, Debezium, Postgres, Centrifugo, Go webserver, and k8s), the frontend remained our Achilles' heel, especially its build process. Despite my seven years of React experience, it still occasionally drives me crazy.

The Frontend Fatigue

Let's talk about our frontend journey - it's quite a tale:

• We needed Babel just to write code with new ES specs

• Webpack became our daily wrestling partner

• Node.js runtime version upgrades felt like walking through a minefield

• Abandoned projects kept us up at night

• Deprecated libraries became our regular headache

• Security issues? Don't get me started

And let's not forget how OOP implementation in JavaScript was a mess back then. In my opinion, most issues in the JS ecosystem stem from the language's inherent fragility – it created an environment where mistakes could slip by unnoticed.

The Revelation

This realization led me to reconsider JavaScript's original purpose: a lightweight scripting language to enhance HTML's user experience, bridging the gap between solid backend-rendered HTML and the browser's flexibility. Maybe it was time to put JavaScript back where it belonged – on the client side, in a more focused role.

Here's what this looks like in practice:

Before (The React Way):

const UserProfile = () => {
  const [user, setUser] = useState(null);
  const [loading, setLoading] = useState(true);
  // More state management...

  useEffect(() => {
    fetchUser()
      .then(data => setUser(data))
      .catch(err => setError(err));
  }, []);

  // Complex rendering logic...
};

After (The HTMX + Go Way):

<!-- Simple, direct, effective: get user after the page loaded -->
<div hx-get="/api/user" hx-trigger="load">
  <!-- Server sends exactly what we need -->
</div>

Learning HTMX: A Personal Challenge

I took HTMX for a test drive in two projects to overcome the honeymoon phenomenon. I'll be honest - it was challenging at first, mainly because of my background. Remember how I mentioned missing the PHP and jQuery era? That gap became apparent. Early in my career, I was deep in mobile app and game development, working with React and React Native, before transitioning directly into backend development.

So, when it came to HTML-over-the-wire, it felt like learning to write with my left hand. Sending partial HTML and using a dedicated API for form validation? Updating the UI via a backend API? It all felt strange initially. But here's the thing - the more I work with it, the more I understand how it makes sense. It isn't that it's difficult; it's that my thinking was conditioned to use the more complicated way.

Why Go + HTMX Clicks

Here's what I've discovered works brilliantly:

1. One Model to Rule Them All

   • No more juggling between frontend and backend models

   • No more omitting fields before sending to the frontend

   • Everything lives where it should - on the server

2. Go's Superpowers in Frontend Code

   • Imagine writing frontend code with Go's compiler watching your back

   • If it builds, most issues are already caught

   • No more undefined/null/empty string gymnastics

3. Development Joy

   Instead of:

    // Dealing with JavaScript uncertainty
    const userEmail = user && user.email || '';
   

   We get (this is Go templ syntax):

    <span>{ User.Email }</span>
   

   Because static typed language already have the solid default value. User.Email is string so the default value is ““. No more null, undefined, ‘‘. madness.

Real-World Impact

In our production environment, this approach has meant:

• Dramatically simpler deployment process

• Faster feature implementation

• Fewer moving parts to maintain

• Better sleep at night (seriously!)

Looking Forward

This journey has taught me that sometimes, simpler really is better. While this doesn't mean we should abandon React or other frontend frameworks entirely – it depends on your needs – it's shown me a more sustainable path for certain types of applications.

In my next post, I'll dive deep into a real project built with this stack. I'll share the nitty-gritty details:

• How we structured our templates

• Where HTMX really shines

• The challenges we faced and overcame

• Practical patterns we discovered along the way

The web development world keeps evolving, and sometimes evolution means rediscovering what we left behind. Stay tuned for more concrete examples and detailed code walkthrough!

If you've ever tried to replace a string containing special characters in Vim, especially across multiple files, you know it can be a real headache. The usual search and replace commands often fall short, getting tripped up by those pesky special characters.

After much trial and error, I've found a reliable method that makes this task straightforward. Here's how to do it:

1. First, use fzf's live grep to find all occurrences of your string:

    :Rg "w.Header().Set(\"Content-Type\", \"text/html; charset=utf-8\")"
   

   This populates your quickfix list with all matches.

2. Now, here's the key command that does the heavy lifting:

    :cfdo %s/\V\Cw.Header().Set("Content-Type", "text\/html; charset=utf-8")/w.Header().Set("Content-Type", "application\/json")/g
   

   Let's break it down:

   • cfdo: Applies the command to all files in the quickfix list.

   • \V: "Very nomagic" mode, which treats most characters literally.

   • \C: Ensures case-sensitive matching.

   • \/: Escapes forward slashes to avoid confusion.

3. Finally, save all your changes:

    :wa
   

And there you have it! This method reliably replaces your string, special characters and all, across your entire project.

No more wrestling with escape characters or pulling your hair out over missed replacements. With this approach, you can handle even the trickiest of string replacements in Vim with ease.

Give it a try next time you're faced with a challenging find-and-replace task. You might be surprised at how smoothly it goes!

In this guide, we'll walk through setting up two Cloud Run services - a public frontend and a private backend - using Terraform. We'll use Google Artifact Registry to store our Docker images. This setup demonstrates how to create a secure, scalable microservices architecture on Google Cloud Platform.

Prerequisites

• A Google Cloud Platform account

• gcloud CLI installed and configured

• Terraform installed

• Docker installed

• Git installed

Getting Started

1. Clone my demo repository:

    git clone git@github.com:finnng/demo-cloud-run-vpc.git
    cd demo-cloud-run-vpc
   

2. Set up your Google Cloud project:

    export PROJECT_ID=your-project-id
    gcloud config set project $PROJECT_ID
   

3. Enable necessary APIs:

    gcloud services enable run.googleapis.com artifactregistry.googleapis.com compute.googleapis.com
   

4. Create a Google Artifact Registry repository:

    gcloud artifacts repositories create cloud-run-demo --repository-format=docker --location=us-central1
   

Building and Pushing Docker Images

1. Build and push the backend image:

    cd backend
    docker build --platform linux/amd64 -t us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/backend:v1 .
    docker push us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/backend:v1
    cd ..
   

2. Build and push the frontend image:

    cd frontend
    docker build --platform linux/amd64 -t us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/frontend:v1 .
    docker push us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/frontend:v1
    cd ..
   

Configuring Terraform

1. Update terraform.tfvars with your project details:

    project_id     = "your-project-id"
    region         = "us-central1"
    frontend_image = "us-central1-docker.pkg.dev/your-project-id/cloud-run-demo/frontend:v1"
    backend_image  = "us-central1-docker.pkg.dev/your-project-id/cloud-run-demo/backend:v1"
   

2. Initialize Terraform:

    terraform init
   

3. Plan and apply the Terraform configuration:

    terraform plan
    terraform apply
   

Understanding the Configuration

Let's look at some key parts of our Terraform configuration:

VPC and Connector Setup

resource "google_compute_network" "vpc_network" {
  name                    = "cloud-run-network"
  auto_create_subnetworks = false
}

resource "google_vpc_access_connector" "connector" {
  name          = "vpc-con"
  ip_cidr_range = "10.8.0.0/28"
  network       = google_compute_network.vpc_network.name
  region        = var.region
}

This creates a VPC network and a connector, allowing our Cloud Run services to communicate securely.

Backend Service

resource "google_cloud_run_v2_service" "backend" {
  name     = "backend"
  location = var.region

  template {
    containers {
      image = var.backend_image
    }
    service_account = google_service_account.demo_backend_sa.email
    vpc_access {
      connector = google_vpc_access_connector.connector.id
      egress    = "PRIVATE_RANGES_ONLY"
    }
  }

  ingress = "INGRESS_TRAFFIC_INTERNAL_ONLY"
}

Note the egress = "PRIVATE_RANGES_ONLY" setting, which allows the backend to make external API calls while maintaining security.

Frontend Service

resource "google_cloud_run_v2_service" "frontend" {
  name     = "frontend"
  location = var.region

  template {
    containers {
      image = var.frontend_image
      env {
        name  = "BACKEND_URL"
        value = google_cloud_run_v2_service.backend.uri
      }
    }
    service_account = google_service_account.demo_frontend_sa.email
    vpc_access {
      connector = google_vpc_access_connector.connector.id
      egress    = "ALL_TRAFFIC"
    }
  }
}

The frontend service is configured to access the backend securely through the VPC connector.

Testing the Setup

After applying the Terraform configuration, you can test your setup:

1. Get the URLs:

    terraform output frontend_url
    terraform output backend_url
   

2. Open the URL in a web browser. You should see the frontend HTML, that means the frontend is publicly access.

3. To verify the backend is private, try accessing its URL directly (you can find it in the GCP Console). It should not be publicly accessible.

4. Now try to let call https://{frontend_url}/request-backend. This request is call to backend and backend make an external call to https://example.com and return the html. You should see the website content of example.com

Cleaning Up

To avoid incurring charges, remember to destroy the resources when you're done:

terraform destroy

Conclusion

This guide demonstrates how to set up a secure microservices architecture using Cloud Run and Terraform. By keeping the backend private and allowing the frontend to communicate with it through a VPC, we've created a scalable and secure setup.

Remember to always consider security best practices, especially in production environments. Regularly review and update your configurations to maintain cloud security standards.

Happy coding, and enjoy exploring Cloud Run and Terraform!

Most teams have a Slack channel full of alerts that everyone eventually mutes. When something breaks, the relevant message is buried under hundreds of copies of the same timeout warning and a pod that's been crash-looping for days.

The noise is repetitive. The same handful of errors repeat, with a timestamp and a pod name swapped in. Repetitive data is compressible, and "group the near-duplicates and show one of each" is a clustering problem with well-established solutions.

I built Alligator, a Slack app that reads a channel over a time range and returns a report that collapses 2,000 alerts into the handful of distinct problems behind them. The Slack integration is the presentation layer. This post is mostly about the ML.

Why not an LLM

Feeding a few thousand log lines to an LLM to summarize the errors works, but it costs money per run, sends production logs to a third party, and returns a different answer each time. The task here is deduplication, and classic ML handles it well: TF-IDF to turn text into vectors, DBSCAN to group the vectors. Both run locally in a couple of seconds, cost nothing, and are deterministic.

The ML pipeline

Take the text of every message in the channel, turn each one into a vector, then cluster the vectors so near-identical messages group together. Three steps: vectorize, cluster, read off the groups.

Step 1 — Text into vectors with TF-IDF

Clustering works on numbers, not strings. TF-IDF (Term Frequency–Inverse Document Frequency) converts a set of documents into vectors:

• Term frequency weights words by how often they appear in a given message.
• Inverse document frequency down-weights words that appear in most messages. If every line contains error, then error carries no information about which cluster a message belongs to.

The result is that the terms distinguishing one kind of alert from another get the most weight, which is what clustering needs.

The implementation is a few lines with scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer

def vectorize_logs(logs):
    vectorizer = TfidfVectorizer(max_features=1000)
    return vectorizer.fit_transform(logs)

Preprocessing: the standard pipeline and what this problem needs

The standard NLP preprocessing pipeline has five steps:

1. Lowercase, so Timeout and timeout count as the same word.
2. Strip punctuation, so timeout, and timeout match.
3. Tokenize, splitting each string into individual words.
4. Remove stop words: high-frequency filler like the, is, a.
5. Stem, reducing words to their root so running, runs, and ran collapse to run.

TfidfVectorizer performs steps 1–3 by default: it lowercases, strips punctuation, and tokenizes on word boundaries with no extra code. This project relies on those defaults and skips steps 4 and 5.

Stop-word removal targets English prose. Log lines are not prose; they are strings like CrashLoopBackOff, connection refused, and context deadline exceeded. Standard stop-word lists rarely match anything in this data, and the terms that actually saturate it (error, warning, failed) are already down-weighted by IDF. A stop-word pass would be redundant here.

Stemming helps when matching natural language that inflects, such as running versus ran. Log lines repeat near-verbatim and do not inflect, so stemming adds a dependency and risks mangling identifiers (Booting and booted both collapse to boot) with no gain in cluster quality.

The point is to run the full pipeline against the data at hand and keep the steps that improve results. A log clusterer and a sentiment classifier start from the same five steps and keep different ones.

max_features=1000 caps the vocabulary at the 1,000 highest-signal terms. Logs have a long tail of unique tokens, since every UUID and timestamp is new, and without a cap the vectors grow to tens of thousands of low-value dimensions. Capping keeps clustering fast and the vectors focused on terms that recur.

The output is a sparse matrix, one row per message and one column per term. A given alert contains only a handful of the 1,000 terms, so storing the zeros would be wasteful. scikit-learn keeps the matrix sparse end to end, which is part of why this is fast.

One domain-specific step would help more than any textbook one: normalizing log tokens before vectorizing. Right now pod-7f8a9 and pod-3b2c1 are distinct tokens, which pushes otherwise-identical alerts apart. Masking the dynamic parts (timestamps to <TS>, UUIDs to <UUID>, numbers to <NUM>) collapses them into one token and tightens every cluster. It is domain-specific, which is why it is not on the generic list.

Step 2 — Grouping with DBSCAN

With the vectors ready, the next step is grouping them. k-means is the common default and is a poor fit here for two reasons.

First, it requires the number of clusters up front. The number of distinct problems in the channel is the unknown the report is meant to surface. Setting k=5 when there are 12 problems merges unrelated errors.

Second, k-means assigns every point to a cluster. DBSCAN does not. It is density-based: it grows a cluster only where points are packed closely together, and points without enough close neighbors are labeled noise (-1).

For this problem the noise label is useful output, not discarded data. The 2,000 repeated timeouts form a dense cluster that can be collapsed. A single panic: nil pointer dereference that appeared once has no dense neighborhood, so it lands in noise. That singleton is the rare, un-triaged event worth a human's attention. DBSCAN separates the repetitive bulk from the one-offs.

The clustering call:

from sklearn.cluster import DBSCAN

def cluster_logs_to_labels(X, eps=0.5, min_samples=5):
    db = DBSCAN(eps=eps, min_samples=min_samples, metric="cosine", n_jobs=-1).fit(X)
    return db.labels_

Three parameters carry the weight:

• metric="cosine". Euclidean distance is the wrong choice for TF-IDF vectors. A long log line and a short one that mean the same thing sit far apart in Euclidean space because one has more words. Cosine distance measures the angle between vectors, comparing which terms a message emphasizes rather than its length. For text, cosine is usually the right metric.
• eps=0.5. The distance threshold for two points to count as neighbors. With cosine distance the range is 0 (identical) to 1 (nothing in common). Lower values produce stricter, more numerous clusters; higher values merge everything into one. 0.5 worked on this data: loose enough to catch the same error across different pods, tight enough to keep distinct problems apart.
• min_samples. How many neighbors a point needs to anchor a cluster. Production runs at 3, so an alert has to fire at least three times to count as a recurring pattern. The benchmark uses 5. This is the first parameter to adjust when tuning.

n_jobs=-1 uses all available cores.

The rest is bookkeeping: walk the labels, drop each message into its cluster, and send everything labeled -1 to the noise list.

clusters = {}
noise = []

for label, log in zip(labels, logs):
    if label == -1:
        noise.append(log)
    else:
        clusters.setdefault(str(label), []).append(log)

That is the ML core: two scikit-learn calls and a loop. No training run, no GPU, no model to host, no API key. It processes a channel in seconds and returns the same result every time.

Wiring Python into a Go service

The app is written in Go, which suits a small concurrent HTTP service. The ML lives in Python because that is where the mature libraries are. The two connect without a microservice, gRPC, or a second container. The Go service runs the Python script as a subprocess and pipes JSON over stdin and stdout.

cmd := exec.Command(pythonPath, pythonScriptPath)
cmd.Stdin = bytes.NewBuffer(inputJSON)

var out, stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr

if err := cmd.Run(); err != nil {
    return nil, fmt.Errorf("error running Python script: %v\nStderr: %s", err, stderr.String())
}

var output ClusterResult
if err := json.Unmarshal(out.Bytes(), &output); err != nil {
    return nil, err
}

The Python side reads JSON from stdin, writes JSON to stdout, and sends all logging to stderr so it never corrupts the data channel:

input_data = json.loads(sys.stdin.read())
logs = input_data["logs"]

X = vectorize_logs(logs)
labels = cluster_logs_to_labels(X, input_data.get("eps", 0.5),
                                   input_data.get("min_samples", 5))

# progress and debug info goes to stderr, results go to stdout
print(f"Processing {len(logs)} logs", file=sys.stderr)
json.dump(output, sys.stdout)

The workload is a batch job that runs for a few seconds and exits. A subprocess with a JSON contract is simple to reason about and to test (echo '{...}' | python tfidf_dbscan.py), and it ships both languages in one container, with the Go binary and Python venv side by side. No service mesh, no second deploy.

Why the split, and why start in Go

The project had two goals: solve the alert-noise problem, and understand TF-IDF and DBSCAN well enough to implement them. So I started in Go and wrote both algorithms by hand.

That path taught me the fundamentals. Implementing TF-IDF and DBSCAN from scratch forces you to work through the sparse vectors, the distance metric, and the neighborhood expansion rather than treat them as a library call. It also stalled. Go has limited tooling for vector and matrix computation, so I was building the numeric primitives myself, and the implementation was dense, naive, and single-threaded. Processing about 4MB of logs took nine minutes. The equivalent Python runs in seconds.

At that point I had what I wanted from writing it by hand. The remaining work was optimization that scikit-learn already does: its TF-IDF and DBSCAN are years of tuned, vectorized, sparse-matrix-aware C behind a Python API. Rewriting that in Go was not the goal; solving the problem was. Switching the ML to Python did both, so the project succeeded on both counts.

For numerical and ML work, the library ecosystem often matters more than the language. Python's libraries are fast because the scientific-computing community has spent decades optimizing them.

The Slack app

The Slack integration is thin. The flow:

1. A user triggers Alligator in a channel. The app opens a modal, built with Slack's Block Kit, asking for a start and end time.
2. On submit, the service fetches the channel history for that window via conversations.history.
3. Clustering takes a few seconds and Slack expects a response within three, so the service returns a link to a report that is not ready yet and runs the work in a background goroutine.
4. When clustering finishes, the report is rendered to HTML and written to a Cloud Storage bucket. The user opens the link to see the clusters.

The respond-now, compute-later split is the part of the Slack integration worth reusing. Slack's 3-second timeout means anything slower than trivial has to run asynchronously:

// respond immediately with a link to the report, which is not ready yet
api.PostEphemeral(channelId, userID, slack.MsgOptionText(
    fmt.Sprintf("Processing analytic. Checkout the report at %s/reports/%s",
        hostURL, reportKey), false))

// run the clustering in the background
go func() {
    output, err := analytic.RunClustering(converted, 0.5, 3)
    if err != nil {
        slog.Error("Error analyzing texts", "error", err)
        return
    }
    report.RenderReport(reportKey, *output)
}()

The report page is server-rendered with templ. It sorts clusters by size so the loudest problems appear first, shows one sample per cluster with the rest behind a collapsible, and lists the noise separately.

Deployment

The service runs on Cloud Run, deployed from master. A push triggers Cloud Build, which builds the single Go and Python image, pushes it to Artifact Registry, and Cloud Run rolls it out with zero downtime. Reports live in a Cloud Storage bucket with a 7-day lifecycle so old ones expire on their own.

Takeaways

• For deduplication, TF-IDF and DBSCAN beat an LLM on cost, speed, privacy, and reproducibility. The task defines the tool.
• DBSCAN's noise label is useful output for triage. An algorithm that forces every point into a cluster discards the rare one-off, which is often the message worth reading.
• For numerical work, the library ecosystem is a first-order concern. The nine-minutes-to-seconds difference came from scikit-learn's optimized C, not from the choice of Python.
• A subprocess with a JSON contract kept two languages in one container with no orchestration. It is worth replacing only when it causes a concrete problem.

What's next

The highest-value change is log-token normalization: masking timestamps, UUIDs, and numbers before vectorizing to tighten clusters. After that, feeding each cluster rather than the full stream to an LLM for a one-line summary. Cheap, deterministic ML compresses 2,000 lines into a handful of groups, and the model only sees the groups, which keeps the cost low.

Beyond that: larger time windows, better monitoring, and a smoother install for the Slack marketplace.

As software engineers, our day-to-day tasks are mostly problem-solving. Given a problem, we find the solution. We gather a lot of skills and practice, and time after time, we become proficient with it. We automatically follow some frameworks but don't realize it.

One of the challenges of an organization is not to depend on the key person, that person who has gathered a lot of knowledge and practice. He can resolve the problem easily; he's a superhero, a firefighter. But if he leaves the company, all of that is lost.

Documentation won't help much; it will be outdated and no one will look at it.

IMO, I think we should focus on building the process, not the key person.

A well-defined process can help the newcomer go through a checklist with only the basic, fundamental knowledge about the system. In case of an emergency, a team member without experience can try the process for the first time, it works for them and if anything is missing from their perspective they can add it to the process.

To be more specific, we call the process a framework. The framework is opinionated by the creator. A place where we can see how the top performers work, extract their methodologies, turn them into a process, and make the excellent a standard for everyone.

Example: Framework to Investigate a Bug

If we have a bug report that something went wrong in the last 3 months, what should we do?

1. Understand the problem, and read the problem statement at least twice to ensure the reported issue is correct

2. If the resolver lacks knowledge, find the help page, find the document, or consult another team member to understand how it works

3. Define the clear expectation

4. Dig into the log, understand why the system doesn't work as expected

5. Dig into the code, read it, and find the wrong logic

6. If the code looks right, maybe it has changed during the last 3 months but gets reverted. Let's check the git logs.

7. Find the ticket related to the code changes to find out what we expected at that time

8. Find the Slack message around that day to discover what is the decision making that didn't write to the ticket properly.

Framework to Implement a New Feature

1. Understand the requirement, and read the problem statement at least twice to ensure the reported issue is correct

2. Define the acceptance criteria

3. Look at the design (if any) twice, and notice all the details missing in the requirement

4. Ask the ticket creator all the questions to clarify the requirement until you make sure all the detail is covered

5. Now work on the code base, and the system design to find the solution

6. Write a solution for preview, maybe a list of bullet points, a diagram, or a document on how you resolve the problem

7. Consult the team members, get an agreement on the proposed solution

8. Go to coding, write the test first if your team follows TDD

9. Submit PR, ask your team for review

10. Resolve feedback, the rest is your normal deployment process...

Let’s start

Implementing problem-solving frameworks isn't just about following a checklist – it's about fostering a culture of shared knowledge and continuous improvement.

Picture this: a team where every member, regardless of experience level, can confidently tackle complex issues. A workplace where knowledge isn't siloed but flows freely, captured in living, breathing frameworks that evolve with each challenge overcome.

This isn't just a pipe dream – it's a reality we can build, one framework at a time.

Here's my challenge to you:

1. Start small: Pick one recurring problem in your team. It could be bug investigation, feature implementation, or even code review.

2. Document your process: Next time you tackle this problem, jot down each step you take. Be specific – what worked? What didn't?

3. Refine and share: Discuss your nascent framework with your team. Gather their insights, refine the steps, and create a shared resource.

4. Iterate and improve: As your team uses the framework, encourage feedback. Let it evolve based on real-world application.

Remember, the goal isn't perfection – it's progress. By taking these steps, you're not just solving today's problems; you're building a foundation for tackling tomorrow's challenges.

I'd love to hear about your experiences. Have you used problem-solving frameworks in your work? What hurdles did you face? What unexpected benefits did you discover? Drop your thoughts in the comments below – let's learn from each other and elevate our craft together.

As a user, I want to set up a time delay for my actions (e.g., send email) in a certain period, a certain day of the week, or a specific day of the year. So that my action can be executed with timing accuracy.

Function requirements

The requirement isn’t strict to any detailed use of the scheduling system. Then I map some concepts into an online marketing system, in which users need to set up a marketing campaign with steps of action running during a period.

• [x] Ensure at least one delivery of every single job

  • [x] Extra: prevent duplicate sends

• [x] The scheduler should be horizontally scalable to handle ~5m timers at peak

• [x] The user can change the value or cancel timers on the fly

• [x] Fairness, priority some group of users

  1. New signed-up users

  2. Tenant with a small audience list.

  3. The rest of the system, but shared the workload between them

• [x] Monitoring metrics Monitoring system

  • [x] Time delay reports end-to-end processing time, from the time the job is due to the time it pulled out of the sending queue to process.

  • [x] Monitor the number of jobs processed per unit of time (e.g., per second)

• [x] TPS: 5m new timers/hour, and 5m due timers/hour

• [x] The scheduler should have a p95 scheduling deviation below 10 seconds

• [x] Cost. The worker didn’t design for scaling another worker internally. I would defer it to a dedicated service like K8S HPA, or self self-managed service that reads the metric and schedules new worker instance

Scope

1. Backend Development: Focus on the backend, interacting with the system via APIs.

2. Scheduling System Implementation:

   • Design and implement a distributed job scheduling system without using/extending any existing Cloud or Open-source job scheduler.

   • Ensure the system can handle 10 million new and/or due timers per hour, with the potential to double this capacity.

   • Include a benchmark to demonstrate the system's capacity.

3. Exclusions:

   • The implementation does not cover the actual job execution; the focus is on the scheduling aspect only. The jobs sent to the execution queue are done.

System design

Core components

1. API server: handle the user requests to schedule the job, handle the job delay time calculation

2. PostgreSQL database: central database, that takes the key role in data consistency.

3. Job processor: the due job checker, the heart of the system. Interval, concurrently check for the due job and send to queue

4. Due job fixer: as the nature of a distributed system the process may stuck due to network errors, or code bugs. The job fixer helps to ensure no job is left behind.

5. Data feeder: Feed the data to the system for demo purposes only

6. Monitoring service. Use an agent to collect metric data for monitoring from the service and workers

The ERD

Entities

I only describe the entities involved in the scope of this project

• Tenant: to get the priority information

• Sequence: The holder of the template for the jobs, including steps and subscribers

• Step: The detailed information to build a job, could be a wait step or a job step

• Subscriber: The subscriber subscribes to the sequence. In this scope, we simply refer to it as a number to count how many jobs we schedule for subscribers of sequence.

• Job: The base unit of the scheduler system, the most important information is status and due time

For the simplicity of the demo, I only define the table for the Job entity

CREATE TABLE IF NOT EXISTS public.jobs
(
    id        serial
        CONSTRAINT jobs_pk
            PRIMARY KEY,
    due_at    timestamp DEFAULT NOW() NOT NULL,
    priority  integer   DEFAULT 0,
    tenant_id integer   DEFAULT 1,
    status    integer   DEFAULT 0,
    metadata  varchar(100)
);

ALTER TABLE public.jobs
    OWNER TO postgres;

CREATE INDEX IF NOT EXISTS jobs_due_at_index
    ON public.jobs (due_at);

CREATE INDEX IF NOT EXISTS jobs_priority_index
    ON public.jobs (priority);

CREATE INDEX IF NOT EXISTS jobs_status_index
    ON public.jobs (status);

Workflows

Users send a POST request to the API server and the system will auto-start the rest of the workflow. Below are the key workflows

1. Schedule job process

Example request

POST <http://localhost:8081/schedule-job>
Content-Type: application/json

{
  "type": "sequence",
  "steps": [
    {
      "type": "wait_certain_period",
      "delay_period": 1,
      "delay_unit": "minute"
    },
    {
      "type": "job",
      "metadata": "{ 'any': 'thing' }"
    },
    {
      "type": "wait_weekday",
      "weekdays": [
        "monday",
        "tuesday",
        "wednesday",
        "thursday",
        "friday"
      ]
    },
    {
      "type": "job",
      "metadata": "job 2"
    },
    {
      "type": "wait_specific_date",
      "date": "2023-12-29T18:48:34.200Z"
    },
    {
      "type": "job",
      "metadata": "job 3"
    }
  ],
  "subscribers": 20
}

To satisfy the design requirement. The schedule-job API accepts a sequence of steps, if the step is a wait step, we will calculate the relative due time, this due time will be the start of the next step in the step list.

No matter where the job step is in the step list, once we get into the calculation of the due time for the job, we only need to use the due time calculated by the previous step as the reference.

Refer to this unit test to understand the expected input and output

func TestCalculateNextJobs(t *testing.T) {
    // Define a start time for the sequence
    startedAt := time.Date(2023, 12, 28, 12, 0, 0, 0, time.UTC)

    // Setup Sequence with steps
    sequence := entity.Sequence{
        Steps: []entity.Step{
            &entity.StepWaitCertainPeriod{DelayPeriod: 1, DelayUnit: entity.DelayUnitMinute},
            &entity.StepJob{Metadata: "{ 'any': 'thing' }"},
            &entity.StepWaitWeekDay{WeekDays: []entity.WeekDay{entity.Monday, entity.Tuesday, entity.Wednesday, entity.Friday}},
            &entity.StepJob{Metadata: "job 2"},
            &entity.StepWaitSpecificDate{Date: "2023-12-29T18:48:34.200Z"},
            &entity.StepJob{Metadata: "job 3"},
        },
        Subscribers: 2,
    }

    // Expected due dates for jobs
    expectedDates := []time.Time{
        startedAt.Add(1 * time.Minute),                           // 1 minute from startedAt (Job 1)
        time.Date(2023, 12, 29, 12, 1, 0, 0, time.UTC),           // Next weekday (Friday) for Job 2
        time.Date(2023, 12, 29, 18, 48, 34, 200000000, time.UTC), // Specific date for Job 3
    }

    got, err := controllers.CalculateNextJobs(sequence, startedAt)
    if err != nil {
        t.Fatalf("CalculateNextJobs() error = %v", err)
    }

    if len(got) != len(expectedDates) {
        t.Fatalf("Expected %d jobs, got %d", len(expectedDates), len(got))
    }

    for i, job := range got {
        log.Print(i, job)
        if !job.DueAt.Equal(expectedDates[i]) {
            t.Errorf("Job %d due at %v, want %v", i, job.DueAt, expectedDates[i])
        }
    }
}

After the job calculation, it will be sent to the database for the bulk insert.

1. Due job checker

After the jobs are inserted into the database with the due date, the due-job-checker workers can pull it from the database.

I use Postgres advisory lock here to ensure only one worker can pull the job at the same time.

But I still use the SELECT FOR UPDATE here to ensure no other process updates my records while I select it.

rows, err := conn.Query(`
              UPDATE jobs 
              SET status = $1
              WHERE id IN (
                  SELECT id FROM jobs 
                  WHERE due_at <= NOW() AND status = $2
                  ORDER BY priority 
                  LIMIT $3
                  FOR UPDATE SKIP LOCKED
              )
              RETURNING id, due_at`, entity.JobStatusInProgress, entity.JobStatusInitialized, dueJobBatchSize)

The lock is released early right after I update its status to processing, this will help to increase the throughput of the system while other incoming actions don’t rely on the database.

1. Due job fixer

To keep the queue table tidy. We need to clean up it frequently. This worker will internally (15 seconds) archive the processed job, in a real system I would move it to the archived table, but in this demo I simply just delete it.

For some reason, if the job fails to process after the maximum processing time limit (let’s say 10s), it is considered a failed job and moved back to the queue.

It is out of scope so I don’t go further on this but in a real system, we need to check more criteria to decide if a job has failed and retry it, not just the exceeded processing time criteria.

Monitoring system

I use external tools for monitoring. Prometheus to collect the metrics and Grafana to plot the data. The tools defined in the docker-compose.yml file

The general idea is to use the Prometheus SDK to collect the metric and send it to the Prometheus push gateway. Then set the Grafana to select the data source.

The metrics I collect

• Job process TPS. This is the number of jobs this system can process per second. As you can see in the screenshot below, the TPS average is ~20.000 jobs/second.

  According to the requirement of 5M new timers + 5M due timers = 10M jobs/hours.

  It can be doubled at any time = 20M jobs/hours

  So 20.000 jobs/sec 60 60 = 72.000.000 jobs/hour. This design can cover the system requirement more than 3 times.

  Note that this system runs on my local machine (MacBook M2 pro base edition), the Postgre database runs in docker with some resource constraints so the real system on the cloud can be much more capable of the higher load.

  

• The p95 of the job due time delay

  An average of 95% of job processing time is 1148 ms. Calculate from the exact UTC that the job due, to the time it is marked as processed.

  This metric is the delayed time from the time the user expected it to be sent to the time it is sent.

  

• Job in queue

  In my data feed, I set it up to schedule a job now and another job delayed 1 min later. As you can see the system is processing both new timers and due timers. As the data feed keeps running every second, this chart won’t be low unless I stop the data feed. But we can expect every job will be processed in about 1 second.

  

Design decision

• Why Go

  • I need the robustness of the language and its ecosystem

  • Foolproof, not afraid of making code mistakes, many good practices and standards can be found online

  • Concurrency model.

• Why PostgreSQL

  • Mostly a silver bullet for most projects that need robustness and flexibility. For example

    • I utilized advisory lock, select for update skip locked for this project

    • In another project, I utilize the jsonb query and view.

  • Rich documentation and resources

Extending ideas

• If I have more time on this project I will refactor it following the Clean Architecture structure to avoid logic fragmenting in many services workers. But for the scope of this project, the structured method works fine.

• Add more integration tests and load tests for all the critical functions. I already have some tests but it is not enough to ensure the robustness of this system. Especially in the concurrent environment.

Read me

Repository

https://github.com/finnng/job-scheduling-system

Prerequisites

• Go version 1.21.0

• docker-compose (you may need to update docker-compose.yml for intel based computer)

Steps

1. Pull the source code

2. Start the databases: docker compose up -d

3. Create a Postgre test database. Use any database client to create a database name test and grant permission for the default user postgre on it.

CREATE DATABASE test;
GRANT ALL PRIVILEGES ON DATABASE test TO postgre;

1. Start the API server, it should automatically provision the tables. From the repo’s root directory, type

go run api-server/app.go

1. Start other workers to complete the full system, open other terminal tabs for these commands

go run worker-due-job-checker/app.go

go run worker-job-fixer/app.go

go run data-feed/app.go

The data-feed worker will randomize the test data and send it to the API server to keep the system busy for the demo purpose. You can edit the test request to test all the cases of the scheduling scenario.

payload := Payload{
            Type: "sequence",
            Steps: []Step{
                {
                    Type:     "job",
                    Metadata: "{ 'any': 'thing 1' }",
                },
                {
                    Type:        "wait_certain_period",
                    DelayPeriod: 1,
                    DelayUnit:   "minute",
                },
                {
                    Type:     "job",
                    Metadata: "{ 'any': 'thing 2' }",
                },
            },
            Subscribers: rand.Intn(10000) + 1, // Random number between 1 and 1000
        }

Your terminal panels should look like this.

Monitoring

I haven’t handled the Grafana database migration yet, so you need to head to the Grafana dashboard at http://localhost:3000 according to the docker-compose file Grafana port.

1. Setup Prometheus as the data source

2. Play around with the metrics sent from the scheduling system

The complete dashboard should look like

I took a basic course on how to create an AI algorithm, it is a Gradient Descent algorithm and it is so basic. I didn't go further with the more advanced algorithms because it is not my expertise area, but it gave me a big picture of how AI works. Prove me wrong. AI now is all about statistics on the highest possible output based on the input.

As long as AI is still based on the training data and the probability, it still needs the input to produce output, it won't be able to destroy any of humankind, and it can't "think" on its own. So what can it do?

For me, I use AI for those purposes

1. Coding

   1. Code suggestion

   2. Code fixing

   3. Chat with code.

   4. Chat with code base (not yet, still waiting for GitHub Enterprise to release this feature - this is freaking cutting edge)

2. Research

   1. Anything, from the new plant I found on the road to the software design philosophy or rocket science

   2. Product research

3. Documentation

   1. Fix grammar

   2. Find the missing argument in my document

4. Helping my wife with her work

   1. Refine the requirement

   2. Draw the diagram

5. Making AI-based product

And many more purposes.

Coding

My company subscribes to the business Github Copilot, surely I have access to the IDE plugin for my Neovim. But I still need the chat, Copilot chat didn't have a plugin for Neovim yet, so I have to create my plugin, you know, by utilizing AI to generate the Lua code. I have another post here on a plugin I copilot with AI to create https://finnng.hashnode.dev/quickly-create-a-scatch-in-neovim

I'm working for a company that partners with other enterprises so data security will be the top concern in mind when using AI for enterprise work. All the code related to work needs to be under the API given by the company.

Benefits

• I can focus on ideas, rather than coding. Because I know coding is easy when ideas are clear.

• Write code faster. Copilot is amazing, it can guess what are my next lines with high accuracy. It speeds me up a lot for the repetitive task and also reminds me of the syntax I don't remember.

• Faster experiment. I don't know Lua, I don't know Rust, or I don't have expertise in Python, but in case I need to modify a lib or a snippet of code I can always ask for AI to help.

• Faster onboarding. When the Copilot Chat With Code Base is released. I believe it will be the game changer. In history, we need the expert in the team to tell you where the place to change code to satisfy the new requirement. Now the new commer and all the developers can benefit from this. We still need humans to work on the code, consider the pros and cons of the solution, and think through the system and how the system affects the users - AI can't do that, but at least it can connect the dots to help you gain understanding faster. Especially for the huge code base in the enterprise company.

Drawbacks

• It makes me lazy. I need to be mindful when using AI. I need to keep in mind how it works, and how to judge its result, but sometimes I am still too lazy to do that and just use the output. This is a dangerous habit. Getting your hand dirty, and scratching your head is a part of the process, it helps to develop the brain. We shouldn't let AI take it from us. You can research this, or I can give a few

  • Learning by doing helps students perform better in science

  • Finding hands-on approaches to remote learning

• I changing my working habits. This one comes from laziness, and also many other drawbacks you can list. The solution is mindfulness when using tools.

Researching

ChatGPT Plus has a crucial role in my research process. Google search is too noisy and not focused on the answer makes me imagine the day that Google becomes another Yahoo.

My research process also changed. ChatGPT Plus can access the internet in real time and it is a game changer. It is based on statistics so it can list out the top popular keywords, and terminologies on the thing I research.

After I have the consolidated keywords, articles, and terminologies. I can ask ChatGPT for more details or I can do a Google search myself to prove the output.

Again, we need to be mindful of the results of AI, and do our research. Keep in mind that it is a copilot. Not a pilot.

Documentation

I'm using Notion AI at work, it is under an enterprise license for sensitive data protection. That's why I can only compose the work documentation with it.

I mostly use it for grammar fixing or sometimes review my docs and find the missing piece I didn't mention or unclear.

Once again, it isn't a "can't live with" tool, it helps me work faster but I can't count how many percent faster.

I found Grammarly to be much more useful as I'm not a native English speaker. Grammarly has a MacOS app that can help to fix Grammar everywhere. It has a clear agreement to not access user sensitive data but I'm not sure so I disabled it in Notion to not let it read the company-related document.

Helping my wife with her work

This is where all the AI comes to shine. My wife is not a developer, nor a technical person, so the output of AI should be natural to humans.

I often ask AI to draw the diagram via the PlantUML code and paste it into https://www.planttext.com/ to view the diagram

ChatGPT Plus has the plugin to draw the diagram in the chat too, but it is a paid solution so I'm fine with the PlantUML.

ChatGPT Plus is so advanced at document refining as the GPT-4 model is training on text. I have no comment about this as the whole world is already impressed by it. But if you use the free account with GPT-3.5 it may not good as you expected.

I installed the app and it can take pictures as the input too. Now I can ask it what is this plant in the picture. After a few back-and-forth corrects it can show me the correct answer.

Voice chat with AI

ChatGPT app also has a voice mode to let you chat with AI by voice. I usually use this in the morning walk while my mind still wandering on some research. It makes me feel like a scene in the movie when a man talking with his AI assistant like talking with people. Interesting, right? Now everyone can hire an assistant.

Making AI-based product

I made a few small products, and editor plugins based on AI. But since it tightens with OpenAI and its core competencies are on OpenAI's shoulder, I never going to do business with it. You can read more about this opinion here https://finnng.hashnode.dev/dont-outsource-your-core-competencies

I may share a few projects in the future if I have time.

Conclusion

That's it. Writing a blog is hard, please give it an upvote if you read it through.

I hope it gives you some insights into using AI to improve your daily life.

If I can come in time to tell one thing to my younger self I will tell him all in BTC back in 2012! Just kidding, I will tell him to choose Java or at least C# Asp.net. Don't choose anything that trendy when you are still a novice, j/k again, who knows what the future holds? And we can't time travel.

JS is still my main tech stack on the backend, along with Go, and C#. But on frontend, I don't have a choice but JS. While the backend can change easily, the frontend is different, I'm not a frontend guy so I don't have a passion for refactoring, or rewriting frontend code regardless of whatever the technology it was.

I am also tired of Webpack, frontend build pipeline, and dependencies chain break because the author adopted something new to their repo. I want to stay with the stable version, but then Dependabot says I have some security risks if I stay with some specific version. What should I do but not to stay up-to-date with my dependencies?

It is time to experiment. But if I do serverside rendering I will go back to the day of PHP and the template engine, back to the AJAX and JQuery if I want to make the page not reload on every click. Then I found HTMX and these things make sense

• Why should only <a> and <form> be able to make HTTP requests?

• Why should only click & submit events trigger them?

• Why should only GET & POST methods be available?

• Why should you only be able to replace the entire screen?

By removing these arbitrary constraints, htmx completes HTML as a hypertext

htmx is a library that allows you to access modern browser features directly from HTML, rather than using javascript.

With HTMX I don't have to use JS anymore, so it requires the server to work together to produce updated HTML (no page reload). The communication between the backend and frontend now is HTML, not JSON, so the server is in charge of frontend logic.

I like this idea, based on my experience, putting the business logic on frontend is a mess, the frontend guy tended to work with frontend only, they tried some magic with the backend JSON response to satisfy the business logic and left the mess here. We asked each other many times, where is the bug, frontend or backend. Hence we can easily trace it, but it is still time-consuming.

We should let the frontend guy do what he does best: frontend, styling, animation. The backend guy does the computation, and data formatting.

Enough talking, in the next part, I will start writing an htmx application.

A scratch file is handy, it auto-saves somewhere on your computer and allows you to paste temporary content here to do whatever you want.

In Vim or Neovim you need to open a new buffer with :new or :vnew , then :set filetype {sometype}. Paste the content, then :w to save it somewhere, you also need to name it, what a cumbersome process.

Say no more, this is my Neovim Lua script.

-- Require the fzf-lua plugin
local fzf_lua = require("fzf-lua")

-- Function to open a new scratch buffer with a specific filetype
local function open_scratch_buffer(filetype)
    -- Generate a timestamp for unique file naming
    local date_time = os.date("%Y-%m-%d_%H-%M-%S")
    -- Define the directory where scratch files will be stored
    local scratch_dir = vim.fn.expand("~/.vim/scratches/")
    -- Create the full path for the new scratch file
    local filename = scratch_dir .. date_time .. "." .. filetype

    -- Create the scratch directory if it does not exist
    vim.fn.mkdir(scratch_dir, "p")
    -- Open a new vertical split with the created file
    vim.cmd("vnew " .. vim.fn.fnameescape(filename))
    -- Set the filetype for the new buffer for appropriate syntax highlighting
    vim.api.nvim_buf_set_option(0, "filetype", filetype)
    -- Save the file to create it on disk
    vim.api.nvim_command("write")
end

-- Global function to select a filetype and create a scratch buffer
function _G.select_filetype_and_create_scratch()
    -- Retrieve the list of Vim syntax files from the runtime path
    local syntax_dir = vim.fn.globpath(vim.fn.getenv("VIMRUNTIME"), "syntax/*.vim", false, true)
    local filetypes = {}

    -- Extract filetype names from the retrieved syntax file paths
    for _, filepath in ipairs(syntax_dir) do
        local filetype = filepath:match("syntax[/\\](.+).vim$")
        if filetype then
            -- Add the filetype to the list
            table.insert(filetypes, filetype)
        end
    end

    -- Execute fzf with the list of filetypes
    fzf_lua.fzf_exec(filetypes, {
        prompt = "Filetypes> ",
        actions = {
            -- Action to perform when a filetype is selected
            ["default"] = function(selected)
                open_scratch_buffer(selected[1])
            end,
        },
    })
end

-- Set a key mapping for the function
-- <leader>t will trigger the filetype selection and scratch buffer creation
vim.api.nvim_set_keymap("n", "<leader>t", ":lua _G.select_filetype_and_create_scratch()<CR>", { noremap = true })

What does it do?

1. Get the list of vim supported file types, grep the base name only to make the list of available file types

2. Use fzf.lua API fzf_exec to serve the list, and provide you with a UI for fuzzy searching

3. Select the user's selected file type, open a new buffer to the right, name it the current datetime, and save it. Now you can paste your content here.

Why the auto-save part is important?

Because some commands like :EslintAutoFix won't run on an unsaved buffer.

To use this, you need:

1. You Neovim are using init.lua

2. You need to install fzf-lua. I tried with fzf.vim but no luck, incompatible.

3. Add this script to your init.lua file require("scratch_config") assumed you have this script name scratch_config

4. Benefit

For more information, check my latest update https://github.com/finnng/dotfiles/blob/master/nvim/lua/scratch_config.lua

This is my repo so it may change at any time, in that case you can visit the commit https://github.com/finnng/dotfiles/commit/c09b79cd455e940cdb07ad836bfce2ca255fd2de

Cut to the chase, here is what I do:

1. The $HOME folder will be the single source of truth. I keep all the original files here

2. Use rsync to sync all the files from $HOME to dotfiles Github local repo

3. Commit the files to the origin to backup.

You may have many files so you may have many rsync commands, let pack it into a bash function and add it to your ~/.zshrc

dotsync() {
  rsync -r ~/.config/nvim/* ~/projects/dotfiles/nvim
  rsync -r ~/.config/karabiner/* ~/projects/dotfiles/karabiner
  rsync -r ~/.config/kitty/* ~/projects/dotfiles/kitty
}

My dotfiles repo is public here https://github.com/finnng/dotfiles

There are a lot of benefits to this approach

1. My dotfiles repo is pristine since the file itself is the single source of truth

2. No more linking issues like sometimes you have to force linking with ln -fs

3. Stress-free, no more file conflict, let the rsync command deal with it.

This week, I just finished a task related to AI. It was nothing special, mostly just calling another third-party service to outsource some work. But even in this seemingly mundane task, there were some interesting insights that I would like to share with you.

Small AI businesses nowadays mostly rely on generative AI to process their core business operations. It's an efficient approach, but it also comes with its fair share of risks. Just yesterday, I observed that the OpenAI service was down, which had a direct impact on a feature in our product. Until the service was back up and running, that feature simply didn't work. Thankfully, it was just a small feature with a lot of fault tolerance built in. But it got me thinking, what if an AI business heavily relied on the OpenAI service? They would go down together, potentially causing significant disruptions to their operations.

This situation made me realize the importance of not outsourcing our core competence. It's easy for anyone to build an application around OpenAI without truly understanding how AI works. However, there's a better way. By using an open-source model, we can take control of our AI implementation. We can run it on our own infrastructure, gaining a deeper understanding of its inner workings. We can train and fine-tune the model to better suit our needs, all while ensuring the security and privacy of our customer data. This approach empowers us to be the true owners of our core competency, allowing us to navigate the ever-changing AI landscape with confidence.

In our company, we take a practical approach to AI. While we use AI to improve certain features and content of our product, our main focus is still on the traditional market. Our product is already leading the market, even without AI. However, AI allows us to enhance our offerings and provide more value to our customers. It's important to note that our success in the market is not solely dependent on AI. We have a team of industry experts who drive our success.

Reflecting on my recent experiences with AI and its role in our business, I appreciate the need for a balanced approach. While AI brings great benefits, we must also be aware of the potential risks and maintain our core competencies. By doing so, we can leverage the power of AI to propel our businesses forward while maintaining control.

Normally we - software engineers go to an interview, and they throw a competitive programming problem to us and say, solve it or **** off. They say that is the best way for engineers to show their problem-solving skills. But I found it isn't true. Here we go, I'm not a big influencer so it won't hurt to keep reading this perspective.

Some of my coworkers are good at solving LeetCode problems, and they are good at requirement problem-solving too. But the underlying aspect of this is some of them cannot still

• Explain the technical understanding to the stakeholders in their language.

• Thinking through the entire system on how this new change affects the other parts and the consequences.

• Well-written and verbal English. I'm not talking about the fluency in English here, I worked with the native UK and US coworker, it's their native language but their presentation was not always as clear as I expected. This skill includes the ability to draw diagrams, write the docs, do presentations, and lead meetings.

• Communication and collaboration, delegation, listening skills, low ego, ease to work with, to teach the other in the day-to-day work.

LeetCode problem, somehow is on the other side of the pragmatic problem-solving skills above. I don't understand why the company tests mathematic skills but doesn't use any of those skills in real work. What they need is a person who can transform the product owner thinking into a running feature, while maintaining the harmonic atmosphere in the team. Live long and prosper.

People with mathematic skill is good for their product at some point, I worked with some 10x engineers, and they rock, but they also burned the whole team to the ground lately with their technical-centric thinking, high IQ doesn't always come with high EQ. Sadly I usually find myself in a pro-IQ interview where competitive programming is not my expertise, some time I pass the interview but I have to work with those 10x engineers, that was a terrible and stressful experience.

Finally, with 10 years of experience in this industry, I can say that people with my kind of problem-solving skills still have a place, with high pay and the open road for higher positions, but the company that respects those skills is not popular.

Pieces of Advice? Here is my 2 cents

For engineers, just keep moving, you know you are the unpopular group, and there are always the right places for you.

For companies, let's say EQ and IQ ranged from 0 to 10. You better look for a person who has both IQ and EQ above 5, that's enough. Don't just look at the IQ 10 and ignore the EQ aspect, the low EQ person is a destroyer. Mathematics problem-solving is not everything, you need engineers with pragmatic skills too. The perfect person may still exist if you keep looking, maybe some day you can find them.