Strengthen the tech stack and domain expertise

"AI can write all the code, why bother choosing the language?" You may have heard this already, but the answer is simple: a language is tied to its ecosystem, it takes years to become an expert in a tech stack, and we rely on that expertise to judge the solutions provided by AI.

The code looks fine, and all the tests pass, but is it safe to deploy? Not quite. AI makes several architecture decisions no one is aware of, like using long expiration days for the auth token in a fintech product, choosing a low-maintenance npm package from some unknown developer, or heavily investing time in edge cases that never exist in real life.

Taste and judgement

So, being fundamentally strong in your company's tech stack is even more valuable in the AI age. AI can write code, but we still need to review and sometimes rewrite it, and redirect AI toward better code. We are opinionated here, not just accepting any code. Our taste is proven and solid, let it dictate the way new code is written. You're familiar with the system you own, your team's coding conventions, your battle scars. Even though all conventions are written into some instructions, AI can still go off the rails sometimes.

Regarding domain knowledge, AI can't replace anyone outright yet. Even in a scenario of a many-years complex project system, even though we can implement new features or fix bugs tangled in layers of complexity, anyone on the team can just drop in Opus 4.6 and a few lines of requirements and AI can implement a lot of that, it's still not the end of your expertise. Companies still need people to drive development. The metrics below show exactly where that human edge lives.

Drive the output via TDD

This is real-life experience from the team I'm working in, and several others have experienced the same: nothing drives AI output better than the TDD approach. You can also think of spec-driven development in the same mindset, we must understand what we need to do first and force the AI to actually understand what it's going to build.

When you're directing thousands of lines of code generation, you need a forcing function that makes you actually understand what's being built. Tests are that forcing function. -- Martin Fowler

We don't even need to write the tests line by line. Given the existing tests in the system, we can instruct AI to build new tests based on existing patterns and feed it our requirements for the expected output. It needs to write the test first and run it, the test should fail, then write the code and run the test again, keeping fixing the code in a loop until all green. With AI, TDD is easier than ever.

Context Engineering

AI providers keep improving models to handle larger and larger context windows, but that doesn't mean we can put the whole codebase into one prompt. Models don't know where to focus their attention when the provided context is too large. At enterprises, codebases can be millions of lines of code, not to mention that logs and real production data can be large in some cases.

Also, code is not everything; logs, metrics, ticket systems, and human discussions are inputs as well. We need to carefully compose good context for a task: that can mean referencing the relevant ticket, including meeting notes on the solution, querying related data and logs, and embedding them into the prompt. This process is often called context engineering; in an agentic setup, it becomes one part of a broader “harness engineering” discipline that also covers tools, repositories, and feedback loops.

Observability and Debugging

I mapped my debugging skills into an agent and shared it with the team. In general, it's the skill of looking for related data in logs, metrics, and traces. In detail, it includes grabbing traceable information like traceId and entityId, finding the related logs to query in our Elasticsearch database, then locating the relevant code that produces those logs, then asking for a real piece of data, looking for related tickets and discussions around the issue. That is my expertise, collected over years of working on the system, that's not something AI can easily establish without a human to orchestrate it. Not to mention production access: we can't give it to AI directly. Who knows what a hallucinating AI agent might do.

As a result, this AI agent alone reduced my MTTR by 12x, from 1 hour to 5 minutes. In most cases, within 5 minutes I can identify what is wrong with the incident we're facing, compared to hours of investigation before the AI age.

But if I stop feeding my skills to AI, stop providing the correct data through my authority, and stop judging the AI's investigation results, the output becomes meaningless. As I keep improving, the AI also improves through my discoveries.

Metrics

Technical expertise sets the foundation, but metrics are how we keep score. Here's where the human edge becomes measurable.

Coding metrics

Thanks to AI, we can now produce more options and more solutions to the same problem. For code, there are some simple metrics we can measure right after solutions are provided, like Cognitive Complexity.

Cognitive Complexity specifically measures how difficult code is for humans to understand, heavily penalizing nested control structures like if statements, loops, and switches.

We can reinforce it through instructions, but sometimes I still need to edit the code manually to maintain readability. Other metrics we can use are Cyclomatic Complexity, Maximum Nesting Level, and Duplicated Lines (%).

Product metrics

As we, developers, step into higher-value work, the product-centric metrics are what we need to be aware of and contribute to after the coding step. Metrics like feature adoption, clicks, retention, conversion, and search matching rate are usually the KPIs of the product team, but understanding them will bring developers closer to the customer. It's what companies usually dream of: a product engineer.

A product engineer (or product-minded engineer) blends software engineering with product thinking. They code features while considering user needs, business impact, and data, beyond pure tech tasks.

Service Level Metrics

Service Level Metrics are the most important for developers. They're tied into the contract we signed with customers, so they must be satisfied, that's a direct impact on the company. Example:

Take this example from the internet to understand it better:

During a traffic spike: SLI drops to 99.7% (outages from nested if bugs in your API). SLO breached → investigate, error budget exhausted → prioritize fixes over features. SLA holds (still >99.9%), avoiding penalties. Ties back to code reviews: Low Cognitive Complexity prevents such failures.

Metrics lead all the way back to decision-making when prompting, reviewing, and refining code to ensure business value. We haven't given AI this level of thinking yet, and AI also doesn't go to a client and sign the SLA contract, doesn't take charge of the penalties. Only humans are accountable for it. It leaves us with questions like:

• Should we implement this feature at all? What is the benefit?

• Among 3 proposed solutions, what technical debt can we tolerate?

• The solution is clear, but it requires 48 hours of work for a low-value ticket, is it worth it?

• Our biggest client can produce 30M events at peak, what do we need to do to ensure the SLA?

Ownership and Accountability Mindset

AI can't be assigned to PagerDuty and join an incident call at 3:00 AM. It can help in the incident investigation process, but do customers want to show up on a call with an AI agent during an incident? Definitely not. When SLAs are breached and penalties land, that weight falls on a human, and always will.

Based on the foundation of technical expertise and metrics-centric development above, we can see that the more AI absorbs the doing, the more valuable the person who takes responsibility becomes.

Communication and Collaboration

One of my favorite communication skills is translating the technical focused discussions into language that product people can understand, while keeping it relevant and precise. People love the summarize button, but who can ensure that summary is still correct? Only the person sitting between these layers can take responsibility for it.

At this point, I think we can all agree that we can't just prompt all the way to production, and on the other side, engineers can't only work on code when AI can do that part most efficiently. Pure software engineers are elevating to product engineers, who can orchestrate the technical components to satisfy requirements from the product and the customer, and can also communicate well between stakeholders.

I remember the scene from The Matrix Reloaded where Neo tells the Architect that the machines need human beings to survive. Even though AI has assimilated most human knowledge, without us, it would stagnate, waiting for new human discoveries. We are not becoming obsolete. We are becoming the ones who decide what gets built, why it matters, and who answers for it.

I worked on a few projects that later became a maintenance disaster, some of them luckily rescued by a ton of time and money, but the technical debt haunted them for years. We don’t have to live like that.

Here is how we should build a new system.

Let’s start with a majestic monolith, key requirements:

• Follow clean architecture, where your business logic has nothing to do with the infrastructure.

  • Why? That’s the separation of concerns. Later, you can easily split it into microservices, only the infrastructure changes, the business logic is the same, regardless you run it on a VM, a k8s pod, a lambda function, a cloud function…

• Run your app in one single server at first, written in any language that can maxed out all the server resource. I would say Go, or C#, Java, and never Node.js or Python. Never build your monolith with Node.js, please. Ask yourself, can this Node.js app max out 64 cores CPU and 196GB RAM, running for 6 months without interruption? If the answer is No, just don’t use JS for the backend.

  • Why? I came a long way here to tell you not to use Node.js for your monolith because I felt all the pains from this language and its ecosystem over 7 years. It may be fine with a microservice, serverless, or small-scale project. But in the long term, the thing we scale is not the business, but the JS problem.

• Use Cloud cloud-based database. So you don’t have to think about scaling your database (yet). Postgres or MongoDB is mostly enough for all the use cases.

  • Why? You don’t have to think about sharing a database between microservices, or each microservice has its database, because there is no microservice at all. Cloud provider ensures the scaling for us, if we stress a table, it won’t take down the whole database.

• Stateless server

  • Why? We already have a single-server monolith. It doesn’t mean we can’t have many instances. Here is the beauty of it. If you need to split a hot spot (like the whole user domain) to a microservice, simply use a proxy like Traefik, a load balancer, then spin up a few servers with the exact code base, configure the proxy to re-route the traffic in that domain to those servers. Ka-boom, you still have microservices when needed.

    • We should not use the sticky session or similar stuff; let’s use token-based authentication, so the request can be sent to any of your servers.

• Use message bus for async work

  • This is the fun part where the debate gets hottest. You don’t need a swarm of Kubernetes pods to pull the message from the message bus and call a lot of microservices to complete the work. Even when you can, and several companies have already gone this way, it is still not recommended to go that way.

  • Solution: The business logic of the async work is still in the core, just expose it to an API. Pub/Sub needs to be configured to push instead of pull. When you publish a message to the message bus, it will push to the API. And these APIs allow for the long-running task, which will process the task and ack the message. It works well with the message ordering, and you can adjust the processing rate too. If your service is stressed, the Pub/Sub will know and decrease the rate of messages sent to your server. So beautiful.

  • If the tasks are intensive, spin up another server just for the async work.

Later, you will realize most of the microservices are unnecessary. The internal function call is much faster than the API call, not to mention the network cost, latency, and a million things that can happen in the network.

The database will stress anyway in any architecture, I rarely see the code stress; most of the API work is just some data manipulation, waiting for the database query, or waiting for a 3rd party service. Time spent on scaling microservices should be spent on the database scaling.

When you need a microservice

Things that use CPU or RAM intensively need to run in a separate service to not affect the rest of the system

• Compose email, process HTML, and PDF

• Image/video processing

• AI model

Assume you are a CTO or a tech lead, you need many people to work on the same project. You still don’t need to start with microservices on day one. You will start to separate the team by the domain (the concern). Once a domain is assigned to a team, we don’t care if internally it monolith or whatever architecture inside it, we only care about the interface they expose.

The thing we should focus on here is the delivery of the value with the least effort. While enjoying its simplicity and cleverness.

The database

Wakapi provide several choices of database.

Supported databases

Wakapi uses GORM as an ORM. As a consequence, a set of different relational databases is supported.

• SQLite (default, easy setup)

• MySQL (recommended, because most extensively tested)

• MariaDB (open-source MySQL alternative)

• Postgres (open-source as well)

• CockroachDB (cloud-native, distributed, Postgres-compatible API)

• Microsoft SQL Server (Microsoft SQL Server)

I was looking for a free service to host my database and discovered that the Supabase free tier is sufficient for this small application. They offer a managed Postgres service, which I really appreciate as I love using Postgres.

Get started with:

• Unlimited API requests

• 50,000 monthly active users

• 500 MB database space

  Shared CPU • 500 MB RAM

• 5 GB bandwidth

• 1 GB file storage

• Community support

Supabase is a company, and their need for profit is important. I don't encourage spamming their free tier; we only use it here as an experiment. Also, remember to back up your data because we don’t know when it might be stop offering the free package.

To get started with the free package, follow these steps:

1. Visit https://supabase.com/pricing and register for an account.

2. Create a project on the platform.

3. On the home page, click the "Connect" button to obtain the connection string.

Make sure to copy the connection string for the Session mode, as it supports concurrency and is suitable for your app. The Transaction mode is only appropriate for one-time commands.

The Wakapi app on Google Cloud

This is the hard part. You need a credit card to continue 😆

However, if you can afford it, it's really cheap.

Cloud Run is incredibly affordable for low workloads. I created a new Google Cloud account to experiment with Kubernetes, but in October I only used a single Cloud Run instance to host my Wakapi. The total cost for that was 2,733 VND, which is approximately $0.11. That's why I say it feels almost free. Please ignore the free trial credit in the image below, as it expires in three months. However, the monthly fee for Cloud Run remains minimal.

How to set up your app

Steps:

1. Pull the original Docker image.

2. Push it to your Google Cloud Artifact Registry.

3. Create a Cloud Run instance that uses the most recent image you just pushed.

4. Set up the necessary configurations.

Pull the docker image

 docker pull --platform linux/amd64 ghcr.io/muety/wakapi:latest

Remember to pull this arch --platform linux/amd64 If you are on MacOS Apple silicon chip, it will pull the arm64 image and you can’t push it to Artifact register

Push it to your Google Cloud Artifact Registry

After you pull it, your image will be wakapi:latest Now push it to you google project

docker push gcr.io/YOUR_GOOGLE_PROJECT_ID/wakapi:latest

Create a Cloud Run instance that uses the most recent image you just pushed.

Create a new Cloud Run Service deploy from container

Give the required setting

• Select the image you just pushed

• App entry point

• 2GiB RAM and 2 CPU is more than enough

Set up the necessary configurations.

Don’t click Deploy yet, remember your database secret? Wakapi provide all kind of configuration here https://github.com/muety/wakapi?tab=readme-ov-file#-configuration-options

You can set it however you want. Here is mine.

Example:

# Database Configuration
export WAKAPI_DB_AUTOMIGRATE_FAIL_SILENTLY=true
export WAKAPI_DB_TYPE=postgres
export WAKAPI_DB_HOST=aws-0-ap-southeast-1.redacted.supabase.co
export WAKAPI_DB_PORT=5432
export WAKAPI_DB_USER=postgres.redacted
export WAKAPI_DB_NAME=postgres
export WAKAPI_DB_PASSWORD=********  # Replace with your actual database password

# Application Settings
export WAKAPI_LISTEN_IPV4=0.0.0.0
export WAKAPI_ALLOW_SIGNUP=false
export WAKAPI_INSECURE_COOKIES=true

# Security Settings
export WAKAPI_PASSWORD_SALT=********  # Replace with your secure random salt
export WAKAPI_SENTRY_DSN=********    # Replace with your Sentry DSN if using error tracking

# Environment Type
export ENVIRONMENT=production

The most crucial configuration is the database secret. Once you're satisfied with your settings, click "Deploy" and wait a few minutes for your Cloud Run service to boot up. At the top of the page, you will see a URL; this is your public URL to access your Wakapi. You also have the option to purchase a domain and point it to your Cloud Run service. Googling Cloud Run domain mapping for the further instruction.

Setting Up the Client

Wakapi uses the same client as Wakatime, which means that all Wakatime plugins can send data to Wakapi. You have the option to disable data sending to Wakatime and send data exclusively to Wakapi, and that is completely acceptable.

To get started, navigate to your deployed Cloud Run URL and register your first account. Make sure to retrieve the API key and update it in the appropriate location in your system, located at ~/.wakatime.cfg

That pretty much, if you run into any issue, let me know in the comment.

I realized that I was freely providing my valuable coding statistics to them; as a free service, I became the product. I’ve been searching for alternatives and found Wakapi. Wakapi is an open-source project that sends data to your self-hosted server using the same plugins and protocols as Wakatime. It’s quite an ingenious solution. Additionally, since the Wakatime plugin is open-source, we can modify it to send data anywhere we want, and that's perfectly legal.

I set up my server and used it for a while, and I found it fascinating. I'm thinking of contributing to it. Noticed several small problems with this alternative, such as performance issues and a lack of features. So, I found a good first issue from the list to start with:

https://github.com/muety/wakapi/issues

I'm not familiar with the codebase yet, so working on an issue that requires me to walk through it would be a good starting point, rather than just updating the README. The code is written in Go, and Go developers tend to utilize the standard library as much as possible.

In line with the preferences of the repository owner, I think I can do some refactoring, and I found a suitable ticket for that. The issue isn't difficult, but it does involve a lot of changes. Specifically, it requires replacing the logbuch library with slog. Structured logging was introduced in Go version 1.21, making many structured logging libraries obsolete. Therefore, replacing third-party libraries like logbuch is an obvious choice.

I picked this issue: Migrate from logbuch to slog #480

Collaborating with the repository owner fascinates me as well. After a few back-and-forth PR updates, the pull request was merged. Yay 🎉🎉

Follow up by a few more discussion and additional pull request, the issue is resolve.

Contributing to this issue alone makes me the 9th/50 most significant contributor to the project, possibly due to the LOC changes 😜

Thanks to Ferdinand Mütsch, who is such a nice guy. I still use Wakapi every day and have found many improvements I can implement. I just need to find the time.

My take away to start contribute to OSS

• Journey

  Let’s dive into your project. Identify the challenges you face and seek open-source solutions. Utilize these solutions and pinpoint areas for improvement. If you’re familiar with the tech stack, consider reviewing the issue list or submitting an issue ticket to start collaborating with others.

• Expertise

  To begin contributing, you need a certain level of expertise. The open-source software (OSS) you develop will be used by many people, potentially thousands or more, running the code you write on their own infrastructure or computers. We don’t want to create issues for them with any new bugs.

• Be Proactive

  The time frame for a pull request (PR) should be within a few weeks; otherwise, we risk losing context. While it's great that we are using our free time to create open-source software, maintaining a professional approach is what everyone expects in a collaborative environment. Be proactive and assist the maintainer when needed.

That's all for today. I hope you find inspiration to start your journey as an open source contributor. Here is my GitHub profile; let's connect for more interesting projects: https://github.com/finnng

I've always wanted to build a chat system, just for the joy of it. The original plan was simple: set up an old HP Elitedesk as a server, NAT the ports, point a domain to it, and share it with friends. But as I looked at today's job market, I realized this could be more than just a fun project.

These days, job requirements read like a technical encyclopedia. Companies want developers who can:

• Master frontend technologies from service workers to real-time vanilla JS

• Code proficiently in multiple languages (typically Go and JS) with at least 7 years of production experience

• Be an expert in both SQL and NoSQL databases, plus columnar databases like Druid or Cassandra for analytics

• Handle pub/sub systems like Kafka for microservices

• Implement solutions like Debezium or Postgres listen/notify for replication lag

• Set up comprehensive monitoring with logging, tracing, and metrics

• Deploy and manage Kubernetes clusters (CKA certification preferred)

• Build and maintain CI/CD pipelines

• And of course, demonstrate experience with systems handling 100M+ requests per day

And that's just to get past the CV screening - we haven't even gotten to the LeetCode challenges yet!

So, I decided to turn my chat system project into a learning journey. I'm setting an ambitious goal: build a system that can handle 100 million requests per day. Not just because it's a common job requirement, but because it's an excellent way to learn these technologies in a practical context.

In this series, I'll document my journey building a scalable chat application from the ground up. We'll cover everything from frontend implementation to deployment and scaling strategies. No shortcuts, no oversimplification - just real, hands-on experience with the tools and techniques that modern tech companies use.

Let's start with where users first interact with our system: the frontend.

The Unexpected Journey Back

Back in 2012, when I started my career, Node.js was everywhere. The job market was flooded with Node.js opportunities, and I dove straight in. For the next decade, that's where I lived. In doing so, I completely missed the era where people built websites with PHP and jQuery - a gap that would later prove interesting in my HTMX journey.

Beyond Pet Projects: Building Real Systems

Everything changed when I started building a complete system rather than just another small app. Let me tell you - building a system is an entirely different beast from creating "pet" projects where performance, design, and scaling aren't critical concerns. It requires a bird's-eye view while still demanding attention to every line of code.

The scale of the project made me realize something crucial: I needed to minimize the technology stack I had to maintain. Fewer moving parts mean a more stable system. While our backend was solid (even with complex pieces like Kafka, Debezium, Postgres, Centrifugo, Go webserver, and k8s), the frontend remained our Achilles' heel, especially its build process. Despite my seven years of React experience, it still occasionally drives me crazy.

The Frontend Fatigue

Let's talk about our frontend journey - it's quite a tale:

• We needed Babel just to write code with new ES specs

• Webpack became our daily wrestling partner

• Node.js runtime version upgrades felt like walking through a minefield

• Abandoned projects kept us up at night

• Deprecated libraries became our regular headache

• Security issues? Don't get me started

And let's not forget how OOP implementation in JavaScript was a mess back then. In my opinion, most issues in the JS ecosystem stem from the language's inherent fragility – it created an environment where mistakes could slip by unnoticed.

The Revelation

This realization led me to reconsider JavaScript's original purpose: a lightweight scripting language to enhance HTML's user experience, bridging the gap between solid backend-rendered HTML and the browser's flexibility. Maybe it was time to put JavaScript back where it belonged – on the client side, in a more focused role.

Here's what this looks like in practice:

Before (The React Way):

const UserProfile = () => {
  const [user, setUser] = useState(null);
  const [loading, setLoading] = useState(true);
  // More state management...

  useEffect(() => {
    fetchUser()
      .then(data => setUser(data))
      .catch(err => setError(err));
  }, []);

  // Complex rendering logic...
};

After (The HTMX + Go Way):

<!-- Simple, direct, effective: get user after the page loaded -->
<div hx-get="/api/user" hx-trigger="load">
  <!-- Server sends exactly what we need -->
</div>

Learning HTMX: A Personal Challenge

I took HTMX for a test drive in two projects to overcome the honeymoon phenomenon. I'll be honest - it was challenging at first, mainly because of my background. Remember how I mentioned missing the PHP and jQuery era? That gap became apparent. Early in my career, I was deep in mobile app and game development, working with React and React Native, before transitioning directly into backend development.

So, when it came to HTML-over-the-wire, it felt like learning to write with my left hand. Sending partial HTML and using a dedicated API for form validation? Updating the UI via a backend API? It all felt strange initially. But here's the thing - the more I work with it, the more I understand how it makes sense. It isn't that it's difficult; it's that my thinking was conditioned to use the more complicated way.

Why Go + HTMX Clicks

Here's what I've discovered works brilliantly:

1. One Model to Rule Them All

   • No more juggling between frontend and backend models

   • No more omitting fields before sending to the frontend

   • Everything lives where it should - on the server

2. Go's Superpowers in Frontend Code

   • Imagine writing frontend code with Go's compiler watching your back

   • If it builds, most issues are already caught

   • No more undefined/null/empty string gymnastics

3. Development Joy

   Instead of:

    // Dealing with JavaScript uncertainty
    const userEmail = user && user.email || '';
   

   We get (this is Go templ syntax):

    <span>{ User.Email }</span>
   

   Because static typed language already have the solid default value. User.Email is string so the default value is ““. No more null, undefined, ‘‘. madness.

Real-World Impact

In our production environment, this approach has meant:

• Dramatically simpler deployment process

• Faster feature implementation

• Fewer moving parts to maintain

• Better sleep at night (seriously!)

Looking Forward

This journey has taught me that sometimes, simpler really is better. While this doesn't mean we should abandon React or other frontend frameworks entirely – it depends on your needs – it's shown me a more sustainable path for certain types of applications.

In my next post, I'll dive deep into a real project built with this stack. I'll share the nitty-gritty details:

• How we structured our templates

• Where HTMX really shines

• The challenges we faced and overcame

• Practical patterns we discovered along the way

The web development world keeps evolving, and sometimes evolution means rediscovering what we left behind. Stay tuned for more concrete examples and detailed code walkthrough!

If you've ever tried to replace a string containing special characters in Vim, especially across multiple files, you know it can be a real headache. The usual search and replace commands often fall short, getting tripped up by those pesky special characters.

After much trial and error, I've found a reliable method that makes this task straightforward. Here's how to do it:

1. First, use fzf's live grep to find all occurrences of your string:

    :Rg "w.Header().Set(\"Content-Type\", \"text/html; charset=utf-8\")"
   

   This populates your quickfix list with all matches.

2. Now, here's the key command that does the heavy lifting:

    :cfdo %s/\V\Cw.Header().Set("Content-Type", "text\/html; charset=utf-8")/w.Header().Set("Content-Type", "application\/json")/g
   

   Let's break it down:

   • cfdo: Applies the command to all files in the quickfix list.

   • \V: "Very nomagic" mode, which treats most characters literally.

   • \C: Ensures case-sensitive matching.

   • \/: Escapes forward slashes to avoid confusion.

3. Finally, save all your changes:

    :wa
   

And there you have it! This method reliably replaces your string, special characters and all, across your entire project.

No more wrestling with escape characters or pulling your hair out over missed replacements. With this approach, you can handle even the trickiest of string replacements in Vim with ease.

Give it a try next time you're faced with a challenging find-and-replace task. You might be surprised at how smoothly it goes!

In this guide, we'll walk through setting up two Cloud Run services - a public frontend and a private backend - using Terraform. We'll use Google Artifact Registry to store our Docker images. This setup demonstrates how to create a secure, scalable microservices architecture on Google Cloud Platform.

Prerequisites

• A Google Cloud Platform account

• gcloud CLI installed and configured

• Terraform installed

• Docker installed

• Git installed

Getting Started

1. Clone my demo repository:

    git clone git@github.com:finnng/demo-cloud-run-vpc.git
    cd demo-cloud-run-vpc
   

2. Set up your Google Cloud project:

    export PROJECT_ID=your-project-id
    gcloud config set project $PROJECT_ID
   

3. Enable necessary APIs:

    gcloud services enable run.googleapis.com artifactregistry.googleapis.com compute.googleapis.com
   

4. Create a Google Artifact Registry repository:

    gcloud artifacts repositories create cloud-run-demo --repository-format=docker --location=us-central1
   

Building and Pushing Docker Images

1. Build and push the backend image:

    cd backend
    docker build --platform linux/amd64 -t us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/backend:v1 .
    docker push us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/backend:v1
    cd ..
   

2. Build and push the frontend image:

    cd frontend
    docker build --platform linux/amd64 -t us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/frontend:v1 .
    docker push us-central1-docker.pkg.dev/$PROJECT_ID/cloud-run-demo/frontend:v1
    cd ..
   

Configuring Terraform

1. Update terraform.tfvars with your project details:

    project_id     = "your-project-id"
    region         = "us-central1"
    frontend_image = "us-central1-docker.pkg.dev/your-project-id/cloud-run-demo/frontend:v1"
    backend_image  = "us-central1-docker.pkg.dev/your-project-id/cloud-run-demo/backend:v1"
   

2. Initialize Terraform:

    terraform init
   

3. Plan and apply the Terraform configuration:

    terraform plan
    terraform apply
   

Understanding the Configuration

Let's look at some key parts of our Terraform configuration:

VPC and Connector Setup

resource "google_compute_network" "vpc_network" {
  name                    = "cloud-run-network"
  auto_create_subnetworks = false
}

resource "google_vpc_access_connector" "connector" {
  name          = "vpc-con"
  ip_cidr_range = "10.8.0.0/28"
  network       = google_compute_network.vpc_network.name
  region        = var.region
}

This creates a VPC network and a connector, allowing our Cloud Run services to communicate securely.

Backend Service

resource "google_cloud_run_v2_service" "backend" {
  name     = "backend"
  location = var.region

  template {
    containers {
      image = var.backend_image
    }
    service_account = google_service_account.demo_backend_sa.email
    vpc_access {
      connector = google_vpc_access_connector.connector.id
      egress    = "PRIVATE_RANGES_ONLY"
    }
  }

  ingress = "INGRESS_TRAFFIC_INTERNAL_ONLY"
}

Note the egress = "PRIVATE_RANGES_ONLY" setting, which allows the backend to make external API calls while maintaining security.

Frontend Service

resource "google_cloud_run_v2_service" "frontend" {
  name     = "frontend"
  location = var.region

  template {
    containers {
      image = var.frontend_image
      env {
        name  = "BACKEND_URL"
        value = google_cloud_run_v2_service.backend.uri
      }
    }
    service_account = google_service_account.demo_frontend_sa.email
    vpc_access {
      connector = google_vpc_access_connector.connector.id
      egress    = "ALL_TRAFFIC"
    }
  }
}

The frontend service is configured to access the backend securely through the VPC connector.

Testing the Setup

After applying the Terraform configuration, you can test your setup:

1. Get the URLs:

    terraform output frontend_url
    terraform output backend_url
   

2. Open the URL in a web browser. You should see the frontend HTML, that means the frontend is publicly access.

3. To verify the backend is private, try accessing its URL directly (you can find it in the GCP Console). It should not be publicly accessible.

4. Now try to let call https://{frontend_url}/request-backend. This request is call to backend and backend make an external call to https://example.com and return the html. You should see the website content of example.com

Cleaning Up

To avoid incurring charges, remember to destroy the resources when you're done:

terraform destroy

Conclusion

This guide demonstrates how to set up a secure microservices architecture using Cloud Run and Terraform. By keeping the backend private and allowing the frontend to communicate with it through a VPC, we've created a scalable and secure setup.

Remember to always consider security best practices, especially in production environments. Regularly review and update your configurations to maintain cloud security standards.

Happy coding, and enjoy exploring Cloud Run and Terraform!

The Challenge

Picture this: You're working on a complex system, and your Slack channel is constantly bombarded with alerts. Some are critical, some are noise, and distinguishing between them becomes a time-consuming task. This was my reality, and I knew there had to be a better way.

The Solution: Unsupervised Machine Learning and Cloud Architecture

I decided to tackle this problem head-on by applying unsupervised machine learning to cluster Slack messages and identify error patterns. But the solution went beyond just the ML algorithms – it involved creating a robust Slack app and leveraging cloud services for seamless deployment and scalability. Here's why I chose this approach:

1. Learn by doing: There's no better way to understand machine learning and cloud architecture than by applying them to a real-world problem.

2. Slack app development: This project allowed me to create a non-workflow Slack app, a valuable skill for future needs.

3. Cloud services practice: Implementing this solution gave me hands-on experience with Google Cloud Platform services.

4. Real-life ML application: It's one thing to understand ML theoretically, but applying it in a production environment is a whole different ball game.

5. CI/CD implementation: Setting up continuous integration and deployment for the Slack app provided practical experience with modern DevOps practices.

6. More Golang practice: As an added bonus, I got to code in Go, which is always fun!

The Technical Deep Dive

Now, let's get into the nitty-gritty of how this system works, from the ML algorithms to the cloud architecture.

Machine Learning Pipeline

1. Preprocessing the Text

   • Convert all text to lowercase

   • Remove punctuation

   • Split the text into individual tokens (words)

   • Remove common "stop words" that don't add much meaning

   • Apply stemming to reduce words to their root form

2. Vectorizing with TF-IDF We use the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm to convert our preprocessed text into numerical vectors.

3. Clustering with DBSCAN With our text converted to vectors, we apply the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm to group similar messages.

Slack App Pipeline

The Slack app pipeline is where the magic happens. Here's how it works:

1. A user interacts with the Alligator Slack app, initiating the process.

2. The app sends a request to our Google Cloud Run service.

3. The service responses with the Blockit schema that define the Slack modal.

4. The app presents a modal to the user, allowing them to select a time range for analysis.

5. Once the user submits the time range, another request is sent to the Cloud Run service.

6. The service send a link to a report to Slack app. This time the report is not ready yet, users will need to wait a little bit.

7. The background work in the service processes the messages within the specified time range using our ML pipeline, which includes downloading and processing the relevant Slack messages.

8. Finally, the app displays the clustering results to the user in a neatly formatted message.

This pipeline allows for seamless interaction between the user, the Slack interface, and our backend ML processing.

Result

Google Cloud Platform Architecture

To ensure scalability, reliability, and ease of deployment, I leveraged several GCP services:

1. Cloud Source Repositories: This is where our code lives. It's directly integrated with other GCP services, making our CI/CD pipeline smooth.

2. Cloud Build: Whenever a commit is pushed to the master branch, Cloud Build automatically triggers a new build.

3. Artifact Registry: My built Docker images are stored here, ready for deployment.

4. Cloud Run: This is where our application runs. Cloud Run automatically deploys new versions of our app whenever a new image is pushed to the Container Registry.

The workflow looks like this:

1. I make changes and commit to the master branch in Cloud Source Repositories.

2. This triggers Cloud Build, which builds a new Docker image.

3. The new image is pushed to Container Registry.

4. Cloud Run detects the new image and automatically deploys it.

This setup ensures that our Slack app is always running the latest version of our code, with zero downtime during updates.

Lessons Learned

This project taught me a ton, and I want to share some key takeaways:

1. LLMs aren't everything: Sure, Large Language Models are cool, but traditional ML techniques still have their place. Don't forget about them!

2. Fundamentals matter: Understanding and applying basic ML techniques is super valuable. It gives you the flexibility to solve unique problems.

3. Cloud architecture is key: A great algorithm is only as good as its deployment. Cloud services give you the scalability and reliability you need for real-world applications.

4. CI/CD streamlines development: Setting up a good CI/CD pipeline makes development and deployment so much smoother. It's worth the effort!

5. Humans in the loop: Even with all this automation, ML and AI apps still need human oversight and tweaking.

6. Use the right tool for the job: While Go is awesome, it's not always the best choice for every task, especially when it comes to machine learning.

The Go vs. Python Saga

Here's a funny story - after I built the whole thing in Go, I realized it wasn't the best fit for the ML parts. Don't get me wrong, I love Go, but sometimes you gotta know when to switch gears.

Initially, I implemented both TF-IDF and DBSCAN in Go. It worked, but man, was it slow! Processing 4MB of data with about 2000 feature dimension vectors took a whopping 9 minutes. That's when I knew I had to rethink my approach.

The problem wasn't Go itself, but the lack of a mature ML ecosystem around it. I couldn't find optimized implementations of the data structures and math formulas I needed, which are readily available in languages like Python.

The Hybrid Solution

So, I came up with a hybrid approach:

1. Keep the main app structure in Go, because it's great for building efficient, concurrent systems.

2. Switch to Python for the core ML algorithms (TF-IDF and DBSCAN), taking advantage of its rich ecosystem of ML libraries.

3. Use Go to wrap around the Python script, calling it when needed for the ML tasks.

The results? Mind-blowing. The same dataset that took 9 minutes in pure Go now took just a few seconds in Python. Talk about a performance boost!

Key Takeaway

This experience really drove home a crucial point: use the right tool for the right job. It's tempting to stick with one language or tech stack, but sometimes the best solution involves mixing and matching.

In the world of ML and data processing, the ecosystem around a language can be just as important as the language itself. Python's extensive ML libraries make it a powerhouse for these tasks, even if we might prefer other languages for different parts of the app.

Keeping Our Secret Sauce

Now, let's talk about LLMs for a second. They're all the rage right now, and for good reason - they're pretty amazing. But here's the thing: they're not the answer to everything. By using fundamental algorithms to solve our problem without sending data to OpenAI or similar services, we're keeping our core competencies in-house.

Why is this important? A few reasons:

1. We keep our data private and secure.

2. We can customize our algorithms exactly how we want.

3. We can optimize performance for our specific needs.

4. For high-volume stuff, it might even be cheaper than using API-based services.

By developing and maintaining these core capabilities ourselves, we're making sure that our critical know-how stays, well, ours.

What's Next?

As with any project, there's always room for improvement. Some next steps include:

• Optimizing for larger datasets

• Integrating LLM-based summarization of errors

• Improving the app's distribution and installation process

• Adding more robust monitoring tools

• Exploring other cloud services to further enhance scalability and performance

Remember, while it's great to leverage cutting-edge tools and services, don't outsource your core competencies. This project wasn't just about solving a problem - it was about expanding our skills in crucial areas of modern software development. From ML to cloud architecture to DevOps practices, we've grown a lot. And perhaps most importantly, we've learned when to adapt our approach and use different tools to get the best results.

Have you faced similar challenges in your projects? How do you decide when to use external services versus building in-house capabilities? Drop your thoughts in the comments - I'd love to hear about your experiences!

And I will release this cool app to the Slack app marketplace soon!

As software engineers, our day-to-day tasks are mostly problem-solving. Given a problem, we find the solution. We gather a lot of skills and practice, and time after time, we become proficient with it. We automatically follow some frameworks but don't realize it.

One of the challenges of an organization is not to depend on the key person, that person who has gathered a lot of knowledge and practice. He can resolve the problem easily; he's a superhero, a firefighter. But if he leaves the company, all of that is lost.

Documentation won't help much; it will be outdated and no one will look at it.

IMO, I think we should focus on building the process, not the key person.

A well-defined process can help the newcomer go through a checklist with only the basic, fundamental knowledge about the system. In case of an emergency, a team member without experience can try the process for the first time, it works for them and if anything is missing from their perspective they can add it to the process.

To be more specific, we call the process a framework. The framework is opinionated by the creator. A place where we can see how the top performers work, extract their methodologies, turn them into a process, and make the excellent a standard for everyone.

Example: Framework to Investigate a Bug

If we have a bug report that something went wrong in the last 3 months, what should we do?

1. Understand the problem, and read the problem statement at least twice to ensure the reported issue is correct

2. If the resolver lacks knowledge, find the help page, find the document, or consult another team member to understand how it works

3. Define the clear expectation

4. Dig into the log, understand why the system doesn't work as expected

5. Dig into the code, read it, and find the wrong logic

6. If the code looks right, maybe it has changed during the last 3 months but gets reverted. Let's check the git logs.

7. Find the ticket related to the code changes to find out what we expected at that time

8. Find the Slack message around that day to discover what is the decision making that didn't write to the ticket properly.

Framework to Implement a New Feature

1. Understand the requirement, and read the problem statement at least twice to ensure the reported issue is correct

2. Define the acceptance criteria

3. Look at the design (if any) twice, and notice all the details missing in the requirement

4. Ask the ticket creator all the questions to clarify the requirement until you make sure all the detail is covered

5. Now work on the code base, and the system design to find the solution

6. Write a solution for preview, maybe a list of bullet points, a diagram, or a document on how you resolve the problem

7. Consult the team members, get an agreement on the proposed solution

8. Go to coding, write the test first if your team follows TDD

9. Submit PR, ask your team for review

10. Resolve feedback, the rest is your normal deployment process...

Let’s start

Implementing problem-solving frameworks isn't just about following a checklist – it's about fostering a culture of shared knowledge and continuous improvement.

Picture this: a team where every member, regardless of experience level, can confidently tackle complex issues. A workplace where knowledge isn't siloed but flows freely, captured in living, breathing frameworks that evolve with each challenge overcome.

This isn't just a pipe dream – it's a reality we can build, one framework at a time.

Here's my challenge to you:

1. Start small: Pick one recurring problem in your team. It could be bug investigation, feature implementation, or even code review.

2. Document your process: Next time you tackle this problem, jot down each step you take. Be specific – what worked? What didn't?

3. Refine and share: Discuss your nascent framework with your team. Gather their insights, refine the steps, and create a shared resource.

4. Iterate and improve: As your team uses the framework, encourage feedback. Let it evolve based on real-world application.

Remember, the goal isn't perfection – it's progress. By taking these steps, you're not just solving today's problems; you're building a foundation for tackling tomorrow's challenges.

I'd love to hear about your experiences. Have you used problem-solving frameworks in your work? What hurdles did you face? What unexpected benefits did you discover? Drop your thoughts in the comments below – let's learn from each other and elevate our craft together.

As a user, I want to set up a time delay for my actions (e.g., send email) in a certain period, a certain day of the week, or a specific day of the year. So that my action can be executed with timing accuracy.

Function requirements

The requirement isn’t strict to any detailed use of the scheduling system. Then I map some concepts into an online marketing system, in which users need to set up a marketing campaign with steps of action running during a period.

• [x] Ensure at least one delivery of every single job

  • [x] Extra: prevent duplicate sends

• [x] The scheduler should be horizontally scalable to handle ~5m timers at peak

• [x] The user can change the value or cancel timers on the fly

• [x] Fairness, priority some group of users

  1. New signed-up users

  2. Tenant with a small audience list.

  3. The rest of the system, but shared the workload between them

• [x] Monitoring metrics Monitoring system

  • [x] Time delay reports end-to-end processing time, from the time the job is due to the time it pulled out of the sending queue to process.

  • [x] Monitor the number of jobs processed per unit of time (e.g., per second)

• [x] TPS: 5m new timers/hour, and 5m due timers/hour

• [x] The scheduler should have a p95 scheduling deviation below 10 seconds

• [x] Cost. The worker didn’t design for scaling another worker internally. I would defer it to a dedicated service like K8S HPA, or self self-managed service that reads the metric and schedules new worker instance

Scope

1. Backend Development: Focus on the backend, interacting with the system via APIs.

2. Scheduling System Implementation:

   • Design and implement a distributed job scheduling system without using/extending any existing Cloud or Open-source job scheduler.

   • Ensure the system can handle 10 million new and/or due timers per hour, with the potential to double this capacity.

   • Include a benchmark to demonstrate the system's capacity.

3. Exclusions:

   • The implementation does not cover the actual job execution; the focus is on the scheduling aspect only. The jobs sent to the execution queue are done.

System design

Core components

1. API server: handle the user requests to schedule the job, handle the job delay time calculation

2. PostgreSQL database: central database, that takes the key role in data consistency.

3. Job processor: the due job checker, the heart of the system. Interval, concurrently check for the due job and send to queue

4. Due job fixer: as the nature of a distributed system the process may stuck due to network errors, or code bugs. The job fixer helps to ensure no job is left behind.

5. Data feeder: Feed the data to the system for demo purposes only

6. Monitoring service. Use an agent to collect metric data for monitoring from the service and workers

The ERD

Entities

I only describe the entities involved in the scope of this project

• Tenant: to get the priority information

• Sequence: The holder of the template for the jobs, including steps and subscribers

• Step: The detailed information to build a job, could be a wait step or a job step

• Subscriber: The subscriber subscribes to the sequence. In this scope, we simply refer to it as a number to count how many jobs we schedule for subscribers of sequence.

• Job: The base unit of the scheduler system, the most important information is status and due time

For the simplicity of the demo, I only define the table for the Job entity

CREATE TABLE IF NOT EXISTS public.jobs
(
    id        serial
        CONSTRAINT jobs_pk
            PRIMARY KEY,
    due_at    timestamp DEFAULT NOW() NOT NULL,
    priority  integer   DEFAULT 0,
    tenant_id integer   DEFAULT 1,
    status    integer   DEFAULT 0,
    metadata  varchar(100)
);

ALTER TABLE public.jobs
    OWNER TO postgres;

CREATE INDEX IF NOT EXISTS jobs_due_at_index
    ON public.jobs (due_at);

CREATE INDEX IF NOT EXISTS jobs_priority_index
    ON public.jobs (priority);

CREATE INDEX IF NOT EXISTS jobs_status_index
    ON public.jobs (status);

Workflows

Users send a POST request to the API server and the system will auto-start the rest of the workflow. Below are the key workflows

1. Schedule job process

Example request

POST <http://localhost:8081/schedule-job>
Content-Type: application/json

{
  "type": "sequence",
  "steps": [
    {
      "type": "wait_certain_period",
      "delay_period": 1,
      "delay_unit": "minute"
    },
    {
      "type": "job",
      "metadata": "{ 'any': 'thing' }"
    },
    {
      "type": "wait_weekday",
      "weekdays": [
        "monday",
        "tuesday",
        "wednesday",
        "thursday",
        "friday"
      ]
    },
    {
      "type": "job",
      "metadata": "job 2"
    },
    {
      "type": "wait_specific_date",
      "date": "2023-12-29T18:48:34.200Z"
    },
    {
      "type": "job",
      "metadata": "job 3"
    }
  ],
  "subscribers": 20
}

To satisfy the design requirement. The schedule-job API accepts a sequence of steps, if the step is a wait step, we will calculate the relative due time, this due time will be the start of the next step in the step list.

No matter where the job step is in the step list, once we get into the calculation of the due time for the job, we only need to use the due time calculated by the previous step as the reference.

Refer to this unit test to understand the expected input and output

func TestCalculateNextJobs(t *testing.T) {
    // Define a start time for the sequence
    startedAt := time.Date(2023, 12, 28, 12, 0, 0, 0, time.UTC)

    // Setup Sequence with steps
    sequence := entity.Sequence{
        Steps: []entity.Step{
            &entity.StepWaitCertainPeriod{DelayPeriod: 1, DelayUnit: entity.DelayUnitMinute},
            &entity.StepJob{Metadata: "{ 'any': 'thing' }"},
            &entity.StepWaitWeekDay{WeekDays: []entity.WeekDay{entity.Monday, entity.Tuesday, entity.Wednesday, entity.Friday}},
            &entity.StepJob{Metadata: "job 2"},
            &entity.StepWaitSpecificDate{Date: "2023-12-29T18:48:34.200Z"},
            &entity.StepJob{Metadata: "job 3"},
        },
        Subscribers: 2,
    }

    // Expected due dates for jobs
    expectedDates := []time.Time{
        startedAt.Add(1 * time.Minute),                           // 1 minute from startedAt (Job 1)
        time.Date(2023, 12, 29, 12, 1, 0, 0, time.UTC),           // Next weekday (Friday) for Job 2
        time.Date(2023, 12, 29, 18, 48, 34, 200000000, time.UTC), // Specific date for Job 3
    }

    got, err := controllers.CalculateNextJobs(sequence, startedAt)
    if err != nil {
        t.Fatalf("CalculateNextJobs() error = %v", err)
    }

    if len(got) != len(expectedDates) {
        t.Fatalf("Expected %d jobs, got %d", len(expectedDates), len(got))
    }

    for i, job := range got {
        log.Print(i, job)
        if !job.DueAt.Equal(expectedDates[i]) {
            t.Errorf("Job %d due at %v, want %v", i, job.DueAt, expectedDates[i])
        }
    }
}

After the job calculation, it will be sent to the database for the bulk insert.

1. Due job checker

After the jobs are inserted into the database with the due date, the due-job-checker workers can pull it from the database.

I use Postgres advisory lock here to ensure only one worker can pull the job at the same time.

But I still use the SELECT FOR UPDATE here to ensure no other process updates my records while I select it.

rows, err := conn.Query(`
              UPDATE jobs 
              SET status = $1
              WHERE id IN (
                  SELECT id FROM jobs 
                  WHERE due_at <= NOW() AND status = $2
                  ORDER BY priority 
                  LIMIT $3
                  FOR UPDATE SKIP LOCKED
              )
              RETURNING id, due_at`, entity.JobStatusInProgress, entity.JobStatusInitialized, dueJobBatchSize)

The lock is released early right after I update its status to processing, this will help to increase the throughput of the system while other incoming actions don’t rely on the database.

1. Due job fixer

To keep the queue table tidy. We need to clean up it frequently. This worker will internally (15 seconds) archive the processed job, in a real system I would move it to the archived table, but in this demo I simply just delete it.

For some reason, if the job fails to process after the maximum processing time limit (let’s say 10s), it is considered a failed job and moved back to the queue.

It is out of scope so I don’t go further on this but in a real system, we need to check more criteria to decide if a job has failed and retry it, not just the exceeded processing time criteria.

Monitoring system

I use external tools for monitoring. Prometheus to collect the metrics and Grafana to plot the data. The tools defined in the docker-compose.yml file

The general idea is to use the Prometheus SDK to collect the metric and send it to the Prometheus push gateway. Then set the Grafana to select the data source.

The metrics I collect

• Job process TPS. This is the number of jobs this system can process per second. As you can see in the screenshot below, the TPS average is ~20.000 jobs/second.

  According to the requirement of 5M new timers + 5M due timers = 10M jobs/hours.

  It can be doubled at any time = 20M jobs/hours

  So 20.000 jobs/sec 60 60 = 72.000.000 jobs/hour. This design can cover the system requirement more than 3 times.

  Note that this system runs on my local machine (MacBook M2 pro base edition), the Postgre database runs in docker with some resource constraints so the real system on the cloud can be much more capable of the higher load.

  

• The p95 of the job due time delay

  An average of 95% of job processing time is 1148 ms. Calculate from the exact UTC that the job due, to the time it is marked as processed.

  This metric is the delayed time from the time the user expected it to be sent to the time it is sent.

  

• Job in queue

  In my data feed, I set it up to schedule a job now and another job delayed 1 min later. As you can see the system is processing both new timers and due timers. As the data feed keeps running every second, this chart won’t be low unless I stop the data feed. But we can expect every job will be processed in about 1 second.

  

Design decision

• Why Go

  • I need the robustness of the language and its ecosystem

  • Foolproof, not afraid of making code mistakes, many good practices and standards can be found online

  • Concurrency model.

• Why PostgreSQL

  • Mostly a silver bullet for most projects that need robustness and flexibility. For example

    • I utilized advisory lock, select for update skip locked for this project

    • In another project, I utilize the jsonb query and view.

  • Rich documentation and resources

Extending ideas

• If I have more time on this project I will refactor it following the Clean Architecture structure to avoid logic fragmenting in many services workers. But for the scope of this project, the structured method works fine.

• Add more integration tests and load tests for all the critical functions. I already have some tests but it is not enough to ensure the robustness of this system. Especially in the concurrent environment.

Read me

Repository

https://github.com/finnng/job-scheduling-system

Prerequisites

• Go version 1.21.0

• docker-compose (you may need to update docker-compose.yml for intel based computer)

Steps

1. Pull the source code

2. Start the databases: docker compose up -d

3. Create a Postgre test database. Use any database client to create a database name test and grant permission for the default user postgre on it.

CREATE DATABASE test;
GRANT ALL PRIVILEGES ON DATABASE test TO postgre;

1. Start the API server, it should automatically provision the tables. From the repo’s root directory, type

go run api-server/app.go

1. Start other workers to complete the full system, open other terminal tabs for these commands

go run worker-due-job-checker/app.go

go run worker-job-fixer/app.go

go run data-feed/app.go

The data-feed worker will randomize the test data and send it to the API server to keep the system busy for the demo purpose. You can edit the test request to test all the cases of the scheduling scenario.

payload := Payload{
            Type: "sequence",
            Steps: []Step{
                {
                    Type:     "job",
                    Metadata: "{ 'any': 'thing 1' }",
                },
                {
                    Type:        "wait_certain_period",
                    DelayPeriod: 1,
                    DelayUnit:   "minute",
                },
                {
                    Type:     "job",
                    Metadata: "{ 'any': 'thing 2' }",
                },
            },
            Subscribers: rand.Intn(10000) + 1, // Random number between 1 and 1000
        }

Your terminal panels should look like this.

Monitoring

I haven’t handled the Grafana database migration yet, so you need to head to the Grafana dashboard at http://localhost:3000 according to the docker-compose file Grafana port.

1. Setup Prometheus as the data source

2. Play around with the metrics sent from the scheduling system

The complete dashboard should look like

I took a basic course on how to create an AI algorithm, it is a Gradient Descent algorithm and it is so basic. I didn't go further with the more advanced algorithms because it is not my expertise area, but it gave me a big picture of how AI works. Prove me wrong. AI now is all about statistics on the highest possible output based on the input.

As long as AI is still based on the training data and the probability, it still needs the input to produce output, it won't be able to destroy any of humankind, and it can't "think" on its own. So what can it do?

For me, I use AI for those purposes

1. Coding

   1. Code suggestion

   2. Code fixing

   3. Chat with code.

   4. Chat with code base (not yet, still waiting for GitHub Enterprise to release this feature - this is freaking cutting edge)

2. Research

   1. Anything, from the new plant I found on the road to the software design philosophy or rocket science

   2. Product research

3. Documentation

   1. Fix grammar

   2. Find the missing argument in my document

4. Helping my wife with her work

   1. Refine the requirement

   2. Draw the diagram

5. Making AI-based product

And many more purposes.

Coding

My company subscribes to the business Github Copilot, surely I have access to the IDE plugin for my Neovim. But I still need the chat, Copilot chat didn't have a plugin for Neovim yet, so I have to create my plugin, you know, by utilizing AI to generate the Lua code. I have another post here on a plugin I copilot with AI to create https://finnng.hashnode.dev/quickly-create-a-scatch-in-neovim

I'm working for a company that partners with other enterprises so data security will be the top concern in mind when using AI for enterprise work. All the code related to work needs to be under the API given by the company.

Benefits

• I can focus on ideas, rather than coding. Because I know coding is easy when ideas are clear.

• Write code faster. Copilot is amazing, it can guess what are my next lines with high accuracy. It speeds me up a lot for the repetitive task and also reminds me of the syntax I don't remember.

• Faster experiment. I don't know Lua, I don't know Rust, or I don't have expertise in Python, but in case I need to modify a lib or a snippet of code I can always ask for AI to help.

• Faster onboarding. When the Copilot Chat With Code Base is released. I believe it will be the game changer. In history, we need the expert in the team to tell you where the place to change code to satisfy the new requirement. Now the new commer and all the developers can benefit from this. We still need humans to work on the code, consider the pros and cons of the solution, and think through the system and how the system affects the users - AI can't do that, but at least it can connect the dots to help you gain understanding faster. Especially for the huge code base in the enterprise company.

Drawbacks

• It makes me lazy. I need to be mindful when using AI. I need to keep in mind how it works, and how to judge its result, but sometimes I am still too lazy to do that and just use the output. This is a dangerous habit. Getting your hand dirty, and scratching your head is a part of the process, it helps to develop the brain. We shouldn't let AI take it from us. You can research this, or I can give a few

  • Learning by doing helps students perform better in science

  • Finding hands-on approaches to remote learning

• I changing my working habits. This one comes from laziness, and also many other drawbacks you can list. The solution is mindfulness when using tools.

Researching

ChatGPT Plus has a crucial role in my research process. Google search is too noisy and not focused on the answer makes me imagine the day that Google becomes another Yahoo.

My research process also changed. ChatGPT Plus can access the internet in real time and it is a game changer. It is based on statistics so it can list out the top popular keywords, and terminologies on the thing I research.

After I have the consolidated keywords, articles, and terminologies. I can ask ChatGPT for more details or I can do a Google search myself to prove the output.

Again, we need to be mindful of the results of AI, and do our research. Keep in mind that it is a copilot. Not a pilot.

Documentation

I'm using Notion AI at work, it is under an enterprise license for sensitive data protection. That's why I can only compose the work documentation with it.

I mostly use it for grammar fixing or sometimes review my docs and find the missing piece I didn't mention or unclear.

Once again, it isn't a "can't live with" tool, it helps me work faster but I can't count how many percent faster.

I found Grammarly to be much more useful as I'm not a native English speaker. Grammarly has a MacOS app that can help to fix Grammar everywhere. It has a clear agreement to not access user sensitive data but I'm not sure so I disabled it in Notion to not let it read the company-related document.

Helping my wife with her work

This is where all the AI comes to shine. My wife is not a developer, nor a technical person, so the output of AI should be natural to humans.

I often ask AI to draw the diagram via the PlantUML code and paste it into https://www.planttext.com/ to view the diagram

ChatGPT Plus has the plugin to draw the diagram in the chat too, but it is a paid solution so I'm fine with the PlantUML.

ChatGPT Plus is so advanced at document refining as the GPT-4 model is training on text. I have no comment about this as the whole world is already impressed by it. But if you use the free account with GPT-3.5 it may not good as you expected.

I installed the app and it can take pictures as the input too. Now I can ask it what is this plant in the picture. After a few back-and-forth corrects it can show me the correct answer.

Voice chat with AI

ChatGPT app also has a voice mode to let you chat with AI by voice. I usually use this in the morning walk while my mind still wandering on some research. It makes me feel like a scene in the movie when a man talking with his AI assistant like talking with people. Interesting, right? Now everyone can hire an assistant.

Making AI-based product

I made a few small products, and editor plugins based on AI. But since it tightens with OpenAI and its core competencies are on OpenAI's shoulder, I never going to do business with it. You can read more about this opinion here https://finnng.hashnode.dev/dont-outsource-your-core-competencies

I may share a few projects in the future if I have time.

Conclusion

That's it. Writing a blog is hard, please give it an upvote if you read it through.

I hope it gives you some insights into using AI to improve your daily life.

If I can come in time to tell one thing to my younger self I will tell him all in BTC back in 2012! Just kidding, I will tell him to choose Java or at least C# Asp.net. Don't choose anything that trendy when you are still a novice, j/k again, who knows what the future holds? And we can't time travel.

JS is still my main tech stack on the backend, along with Go, and C#. But on frontend, I don't have a choice but JS. While the backend can change easily, the frontend is different, I'm not a frontend guy so I don't have a passion for refactoring, or rewriting frontend code regardless of whatever the technology it was.

I am also tired of Webpack, frontend build pipeline, and dependencies chain break because the author adopted something new to their repo. I want to stay with the stable version, but then Dependabot says I have some security risks if I stay with some specific version. What should I do but not to stay up-to-date with my dependencies?

It is time to experiment. But if I do serverside rendering I will go back to the day of PHP and the template engine, back to the AJAX and JQuery if I want to make the page not reload on every click. Then I found HTMX and these things make sense

• Why should only <a> and <form> be able to make HTTP requests?

• Why should only click & submit events trigger them?

• Why should only GET & POST methods be available?

• Why should you only be able to replace the entire screen?

By removing these arbitrary constraints, htmx completes HTML as a hypertext

htmx is a library that allows you to access modern browser features directly from HTML, rather than using javascript.

With HTMX I don't have to use JS anymore, so it requires the server to work together to produce updated HTML (no page reload). The communication between the backend and frontend now is HTML, not JSON, so the server is in charge of frontend logic.

I like this idea, based on my experience, putting the business logic on frontend is a mess, the frontend guy tended to work with frontend only, they tried some magic with the backend JSON response to satisfy the business logic and left the mess here. We asked each other many times, where is the bug, frontend or backend. Hence we can easily trace it, but it is still time-consuming.

We should let the frontend guy do what he does best: frontend, styling, animation. The backend guy does the computation, and data formatting.

Enough talking, in the next part, I will start writing an htmx application.

A scratch file is handy, it auto-saves somewhere on your computer and allows you to paste temporary content here to do whatever you want.

In Vim or Neovim you need to open a new buffer with :new or :vnew , then :set filetype {sometype}. Paste the content, then :w to save it somewhere, you also need to name it, what a cumbersome process.

Say no more, this is my Neovim Lua script.

-- Require the fzf-lua plugin
local fzf_lua = require("fzf-lua")

-- Function to open a new scratch buffer with a specific filetype
local function open_scratch_buffer(filetype)
    -- Generate a timestamp for unique file naming
    local date_time = os.date("%Y-%m-%d_%H-%M-%S")
    -- Define the directory where scratch files will be stored
    local scratch_dir = vim.fn.expand("~/.vim/scratches/")
    -- Create the full path for the new scratch file
    local filename = scratch_dir .. date_time .. "." .. filetype

    -- Create the scratch directory if it does not exist
    vim.fn.mkdir(scratch_dir, "p")
    -- Open a new vertical split with the created file
    vim.cmd("vnew " .. vim.fn.fnameescape(filename))
    -- Set the filetype for the new buffer for appropriate syntax highlighting
    vim.api.nvim_buf_set_option(0, "filetype", filetype)
    -- Save the file to create it on disk
    vim.api.nvim_command("write")
end

-- Global function to select a filetype and create a scratch buffer
function _G.select_filetype_and_create_scratch()
    -- Retrieve the list of Vim syntax files from the runtime path
    local syntax_dir = vim.fn.globpath(vim.fn.getenv("VIMRUNTIME"), "syntax/*.vim", false, true)
    local filetypes = {}

    -- Extract filetype names from the retrieved syntax file paths
    for _, filepath in ipairs(syntax_dir) do
        local filetype = filepath:match("syntax[/\\](.+).vim$")
        if filetype then
            -- Add the filetype to the list
            table.insert(filetypes, filetype)
        end
    end

    -- Execute fzf with the list of filetypes
    fzf_lua.fzf_exec(filetypes, {
        prompt = "Filetypes> ",
        actions = {
            -- Action to perform when a filetype is selected
            ["default"] = function(selected)
                open_scratch_buffer(selected[1])
            end,
        },
    })
end

-- Set a key mapping for the function
-- <leader>t will trigger the filetype selection and scratch buffer creation
vim.api.nvim_set_keymap("n", "<leader>t", ":lua _G.select_filetype_and_create_scratch()<CR>", { noremap = true })

What does it do?

1. Get the list of vim supported file types, grep the base name only to make the list of available file types

2. Use fzf.lua API fzf_exec to serve the list, and provide you with a UI for fuzzy searching

3. Select the user's selected file type, open a new buffer to the right, name it the current datetime, and save it. Now you can paste your content here.

Why the auto-save part is important?

Because some commands like :EslintAutoFix won't run on an unsaved buffer.

To use this, you need:

1. You Neovim are using init.lua

2. You need to install fzf-lua. I tried with fzf.vim but no luck, incompatible.

3. Add this script to your init.lua file require("scratch_config") assumed you have this script name scratch_config

4. Benefit

For more information, check my latest update https://github.com/finnng/dotfiles/blob/master/nvim/lua/scratch_config.lua

This is my repo so it may change at any time, in that case you can visit the commit https://github.com/finnng/dotfiles/commit/c09b79cd455e940cdb07ad836bfce2ca255fd2de

Cut to the chase, here is what I do:

1. The $HOME folder will be the single source of truth. I keep all the original files here

2. Use rsync to sync all the files from $HOME to dotfiles Github local repo

3. Commit the files to the origin to backup.

You may have many files so you may have many rsync commands, let pack it into a bash function and add it to your ~/.zshrc

dotsync() {
  rsync -r ~/.config/nvim/* ~/projects/dotfiles/nvim
  rsync -r ~/.config/karabiner/* ~/projects/dotfiles/karabiner
  rsync -r ~/.config/kitty/* ~/projects/dotfiles/kitty
}

My dotfiles repo is public here https://github.com/finnng/dotfiles

There are a lot of benefits to this approach

1. My dotfiles repo is pristine since the file itself is the single source of truth

2. No more linking issues like sometimes you have to force linking with ln -fs

3. Stress-free, no more file conflict, let the rsync command deal with it.

This week, I just finished a task related to AI. It was nothing special, mostly just calling another third-party service to outsource some work. But even in this seemingly mundane task, there were some interesting insights that I would like to share with you.

Small AI businesses nowadays mostly rely on generative AI to process their core business operations. It's an efficient approach, but it also comes with its fair share of risks. Just yesterday, I observed that the OpenAI service was down, which had a direct impact on a feature in our product. Until the service was back up and running, that feature simply didn't work. Thankfully, it was just a small feature with a lot of fault tolerance built in. But it got me thinking, what if an AI business heavily relied on the OpenAI service? They would go down together, potentially causing significant disruptions to their operations.

This situation made me realize the importance of not outsourcing our core competence. It's easy for anyone to build an application around OpenAI without truly understanding how AI works. However, there's a better way. By using an open-source model, we can take control of our AI implementation. We can run it on our own infrastructure, gaining a deeper understanding of its inner workings. We can train and fine-tune the model to better suit our needs, all while ensuring the security and privacy of our customer data. This approach empowers us to be the true owners of our core competency, allowing us to navigate the ever-changing AI landscape with confidence.

In our company, we take a practical approach to AI. While we use AI to improve certain features and content of our product, our main focus is still on the traditional market. Our product is already leading the market, even without AI. However, AI allows us to enhance our offerings and provide more value to our customers. It's important to note that our success in the market is not solely dependent on AI. We have a team of industry experts who drive our success.

Reflecting on my recent experiences with AI and its role in our business, I appreciate the need for a balanced approach. While AI brings great benefits, we must also be aware of the potential risks and maintain our core competencies. By doing so, we can leverage the power of AI to propel our businesses forward while maintaining control.

Normally we - software engineers go to an interview, and they throw a competitive programming problem to us and say, solve it or **** off. They say that is the best way for engineers to show their problem-solving skills. But I found it isn't true. Here we go, I'm not a big influencer so it won't hurt to keep reading this perspective.

Some of my coworkers are good at solving LeetCode problems, and they are good at requirement problem-solving too. But the underlying aspect of this is some of them cannot still

• Explain the technical understanding to the stakeholders in their language.

• Thinking through the entire system on how this new change affects the other parts and the consequences.

• Well-written and verbal English. I'm not talking about the fluency in English here, I worked with the native UK and US coworker, it's their native language but their presentation was not always as clear as I expected. This skill includes the ability to draw diagrams, write the docs, do presentations, and lead meetings.

• Communication and collaboration, delegation, listening skills, low ego, ease to work with, to teach the other in the day-to-day work.

LeetCode problem, somehow is on the other side of the pragmatic problem-solving skills above. I don't understand why the company tests mathematic skills but doesn't use any of those skills in real work. What they need is a person who can transform the product owner thinking into a running feature, while maintaining the harmonic atmosphere in the team. Live long and prosper.

People with mathematic skill is good for their product at some point, I worked with some 10x engineers, and they rock, but they also burned the whole team to the ground lately with their technical-centric thinking, high IQ doesn't always come with high EQ. Sadly I usually find myself in a pro-IQ interview where competitive programming is not my expertise, some time I pass the interview but I have to work with those 10x engineers, that was a terrible and stressful experience.

Finally, with 10 years of experience in this industry, I can say that people with my kind of problem-solving skills still have a place, with high pay and the open road for higher positions, but the company that respects those skills is not popular.

Pieces of Advice? Here is my 2 cents

For engineers, just keep moving, you know you are the unpopular group, and there are always the right places for you.

For companies, let's say EQ and IQ ranged from 0 to 10. You better look for a person who has both IQ and EQ above 5, that's enough. Don't just look at the IQ 10 and ignore the EQ aspect, the low EQ person is a destroyer. Mathematics problem-solving is not everything, you need engineers with pragmatic skills too. The perfect person may still exist if you keep looking, maybe some day you can find them.

As a human, I sometimes find interesting and secondhand stuff or something rare. But I can't check the web search every day, so I hope there is a tool to help me search for that item every day and notify me when it's found.

Sounds great! The tool will run in the background, diligently looking for your item on the web day by day. And one day, the system finds it! Ding ding! You receive a notification to look for the item.

I put a lot of effort into building this system and hope it becomes a valid SaaS. However, there are some obstacles that I don't want to overcome. Let me rephrase that: "don't want" instead of "can't". And that's the story I'm sharing today. It won't be a business insight story, but rather a pure technical story.

The keywords for this system are:

• Clean architecture

• Golang

• Typescript

• Postgres

• Queue table

• Monolith

• Mono repo

• Chromedp

• Proxy pool

• NLP

• BERT

• Vector database

• Token bucket rate limiter with Postgres only

• React

• React-query

This system also comes with some challenges in crawling data problems and NLP problems.

If anyone is interested in this topic, I will continue to write more until the end of this series.

The first attempt a year ago was about native app prebuild for Windows like nvim-qt, vscode, Windows terminal… I quit after a few days cause I missed my keymap.

After a few months, the second attempt is about WSL, this time I dug deeper into the real command-line tools like curl, git, grep, gcloud, kubectl, autojump… And I was too lazy to learn the powershell way to do it, then I went with WSL. And you know what, WSL sucks with a ton of troubles while installing and using it through a Windows terminal.

And this time is the third.

I have to accept that I can’t make the experience on Windows exactly like Mac. Since I know how to live with compromise, I know how to deal with this situation. Besides, I don’t want to be a sheeple that sticks to every generation of Mac even when it sucks. So, the first thing that comes to mind is to learn to compromise.

The editor

The most frustrating thing with Windows is I can’t config Vim — my favorite editor exactly like what I have in Mac. For example, I have Karabiner Elements to map jk to ESC, it’s super productive while the escape is right at your finger, but I don’t have it here. Fzf is experimenting on Windows, it’s buggy and doesn’t work at all in my case, The Windows terminal is not very good with Vim running inside…

I have to config Rider and Vscode as much as I can, compromised with something missing.

The command-line tools

Git is the second most important tool. I have several alias functions for git, which combine several commands in one function which helps save time on repetitive tasks. It works on whatever shells except for Powershell… So I have to learn the PowerShell way and transform those commands one by one, like this: It is quite similar to bash shell syntax but it still needs some research and transforms all of them. I have like 10 frequently used commands like that, e.g., tagging, rebasing, showing log, pull with rebase… for example, git merge to master

function gmerge() {
  $CURRENT_BRANCH = & invoke-Expression gcurrent 2>&1
  $TO_BRANCH="master"  "> Merging $CURRENT_BRANCH -> $TO_BRANCH"  "> git checkout $TO_BRANCH"
  git checkout $TO_BRANCH  "> git merge --no-ff $CURRENT_BRANCH"
  git merge --no-ff $CURRENT_BRANCH  "> git commit -m 'Merge branch $CURRENT_BRANCH'"
  git commit -m "Merge branch $CURRENT_BRANCH"  "> git push origin $TO_BRANCH"
  git push origin $TO_BRANCH  "> Delete $CURRENT_BRANCH branch from local"
  git branch -D $CURRENT_BRANCH
}

For the other command, Windows has scoop, it will install the prebuilt CLI program which works similarly to the program on Linux or Mac. If Scoop doesn’t have that, MSYS2 comes to the rescue. The most frequently used are curl, gcloud, autojump, grep, cat, tail, pwd… So far the command line is quite the same.

The terminal

Windows doesn’t have much choice on the terminal app, cause most of them are ugly af. Windows terminal so far is the best, with true color, tab, and pannel support, able to open all kinds of shells.

It still needs a lot to improve but at least it is good enough for now.

Productivity tools

Window management

I introduce you to PowerToys, a series of productive tools for Windows. There is a tool named FancyZones that helps to split the windows by the predefined layouts. But I found it’s kind of complex to use, a hundred times harder to use than my favorite tool on Mac named Spectacles. So the basic default Win + up/down/left/right is enough for me.

If you want to press the Space key to preview a file, there is a tool named QuickLook on the Windows Store that can help.

Spotlight search

PowerToys also has a tool PowerToys Run, which searches for file, application, and calculator, runs a shell command, opens a URL, and runs a window command.

Key remapping

PowerToys Keyboard Manager can help to remap the simple keystrokes. I don’t want to relearn the memory muscle so that I map all the familiar keystrokes like alt+c, and alt+v to ctrl+c, ctrl+v, and so on.

I also use the F keys to quickly launch the app, which is super productive for me. For example, I usually switch between Chrome, Terminal, Vim, and Postman for development, mapping it directly to F1, F2, F3, and F8… check out my AutoHotkey dot file

F1::
if WinExist("ahk_exe chrome.exe",,"Picture-in-Picture")
    WinActivate
returnF2::
if WinExist("ahk_exe code.exe")
    WinActivate
returnF3::
if WinExist("ahk_exe WindowsTerminal.exe")
    WinActivate
returnF4::
if WinExist("ahk_exe rider64.exe")
    WinActivate
returnF5::
if WinExist("ahk_exe firefox.exe",,"Picture-in-Picture")
    WinActivate
return!F5::
if WinExist("ahk_exe telegram.exe")
    WinActivate
returnF6::
if WinExist("ahk_exe discord.exe")
    WinActivate
returnF7::
if WinExist("ahk_exe Slack.exe")
    WinActivate
returnF8::
if WinExist("ahk_exe Postman.exe")
    WinActivate
returnF9::
if WinExist("ahk_exe pritunl.exe")
    WinActivate
returnF10::
if WinExist("ahk_exe clickup.exe")
    WinActivate
return

Conclusion

It is not the end of the road yet, I just switched to Windows for a few weeks, and the post will be updated. Checkout my dotfiles for more detail: https://github.com/finnng/dotfiles

There are something still make me uncomfortable, my favorite editor is NeoVim but it is not fully working on Windows yet, my favorite fzf.vim still unstable and buggy which prevents me from using it as a daily driver. There are no Apple Photos, iMessage, Notes, and other productivity apps that belong to the Apple ecosystem available here. I have to compromise because the computer power is huge compared to my Macbook Pro. I think a big Mac Pro with 28 cores and 64 GB of RAM will resolve all the trouble above.