Series Building a Chat System that Scales: A Developer's Journey
Building a Scalable Chat System with HTMX and Go: Lessons from My Journey
A Developer's Journey
I've always wanted to build a chat system, just for the joy of it. The original plan was simple: set up an old HP Elitedesk as a server, NAT the ports, point a domain to it, and share it with friends. But as I looked at today's job market, I realized this could be more than just a fun project.
These days, job requirements read like a technical encyclopedia. Companies want developers who can:
Master frontend technologies from service workers to real-time vanilla JS
Code proficiently in multiple languages (typically Go and JS) with at least 7 years of production experience
Be an expert in both SQL and NoSQL databases, plus columnar databases like Druid or Cassandra for analytics
Handle pub/sub systems like Kafka for microservices
Implement solutions like Debezium or Postgres listen/notify for replication lag
Set up comprehensive monitoring with logging, tracing, and metrics
Deploy and manage Kubernetes clusters (CKA certification preferred)
Build and maintain CI/CD pipelines
And of course, demonstrate experience with systems handling 100M+ requests per day
And that's just to get past the CV screening - we haven't even gotten to the LeetCode challenges yet!
So, I decided to turn my chat system project into a learning journey. I'm setting an ambitious goal: build a system that can handle 100 million requests per day. Not just because it's a common job requirement, but because it's an excellent way to learn these technologies in a practical context.
In this series, I'll document my journey building a scalable chat application from the ground up. We'll cover everything from frontend implementation to deployment and scaling strategies. No shortcuts, no oversimplification - just real, hands-on experience with the tools and techniques that modern tech companies use.
Let's start with where users first interact with our system: the frontend.
The Unexpected Journey Back
Back in 2012, when I started my career, Node.js was everywhere. The job market was flooded with Node.js opportunities, and I dove straight in. For the next decade, that's where I lived. In doing so, I completely missed the era where people built websites with PHP and jQuery - a gap that would later prove interesting in my HTMX journey.
Beyond Pet Projects: Building Real Systems
Everything changed when I started building a complete system rather than just another small app. Let me tell you - building a system is an entirely different beast from creating "pet" projects where performance, design, and scaling aren't critical concerns. It requires a bird's-eye view while still demanding attention to every line of code.
The scale of the project made me realize something crucial: I needed to minimize the technology stack I had to maintain. Fewer moving parts mean a more stable system. While our backend was solid (even with complex pieces like Kafka, Debezium, Postgres, Centrifugo, Go webserver, and k8s), the frontend remained our Achilles' heel, especially its build process. Despite my seven years of React experience, it still occasionally drives me crazy.
The Frontend Fatigue
Let's talk about our frontend journey - it's quite a tale:
We needed Babel just to write code with new ES specs
Webpack became our daily wrestling partner
Node.js runtime version upgrades felt like walking through a minefield
Abandoned projects kept us up at night
Deprecated libraries became our regular headache
Security issues? Don't get me started
And let's not forget how OOP implementation in JavaScript was a mess back then. In my opinion, most issues in the JS ecosystem stem from the language's inherent fragility – it created an environment where mistakes could slip by unnoticed.
The Revelation
This realization led me to reconsider JavaScript's original purpose: a lightweight scripting language to enhance HTML's user experience, bridging the gap between solid backend-rendered HTML and the browser's flexibility. Maybe it was time to put JavaScript back where it belonged – on the client side, in a more focused role.
Here's what this looks like in practice:
Before (The React Way):
const UserProfile = () => {
const [user, setUser] = useState(null);
const [loading, setLoading] = useState(true);
// More state management...
useEffect(() => {
fetchUser()
.then(data => setUser(data))
.catch(err => setError(err));
}, []);
// Complex rendering logic...
};
After (The HTMX + Go Way):
<!-- Simple, direct, effective: get user after the page loaded -->
<div hx-get="/api/user" hx-trigger="load">
<!-- Server sends exactly what we need -->
</div>
Learning HTMX: A Personal Challenge
I took HTMX for a test drive in two projects to overcome the honeymoon phenomenon. I'll be honest - it was challenging at first, mainly because of my background. Remember how I mentioned missing the PHP and jQuery era? That gap became apparent. Early in my career, I was deep in mobile app and game development, working with React and React Native, before transitioning directly into backend development.
So, when it came to HTML-over-the-wire, it felt like learning to write with my left hand. Sending partial HTML and using a dedicated API for form validation? Updating the UI via a backend API? It all felt strange initially. But here's the thing - the more I work with it, the more I understand how it makes sense. It isn't that it's difficult; it's that my thinking was conditioned to use the more complicated way.
Why Go + HTMX Clicks
Here's what I've discovered works brilliantly:
One Model to Rule Them All
No more juggling between frontend and backend models
No more omitting fields before sending to the frontend
Everything lives where it should - on the server
Go's Superpowers in Frontend Code
Imagine writing frontend code with Go's compiler watching your back
If it builds, most issues are already caught
No more undefined/null/empty string gymnastics
Development Joy
Instead of:
// Dealing with JavaScript uncertainty const userEmail = user && user.email || '';
We get (this is
Go templ
syntax):<span>{ User.Email }</span>
Because static typed language already have the solid default value. User.Email is string so the default value is
““
. No morenull
,undefined
,‘‘
. madness.
Real-World Impact
In our production environment, this approach has meant:
Dramatically simpler deployment process
Faster feature implementation
Fewer moving parts to maintain
Better sleep at night (seriously!)
Looking Forward
This journey has taught me that sometimes, simpler really is better. While this doesn't mean we should abandon React or other frontend frameworks entirely – it depends on your needs – it's shown me a more sustainable path for certain types of applications.
In my next post, I'll dive deep into a real project built with this stack. I'll share the nitty-gritty details:
How we structured our templates
Where HTMX really shines
The challenges we faced and overcame
Practical patterns we discovered along the way
The web development world keeps evolving, and sometimes evolution means rediscovering what we left behind. Stay tuned for more concrete examples and detailed code walkthrough!