DEV Community: ude-p

Building a Deployment Platform on Self-Managed Infrastructure with k3s

ude-p — Wed, 24 Jun 2026 19:49:17 +0000

At the beginning of this year, our production workloads were spread across AWS, Contabo and GoDaddy.

Most of the applications had predictable traffic patterns and fairly stable resource requirements. As more products were added, infrastructure costs became harder to ignore, particularly on AWS. The deployment workflows, infrastructure layouts and operational processes supporting those products had also become increasingly inconsistent.

Part of my responsibility became consolidating that environment, reducing unnecessary infrastructure costs and introducing a more consistent deployment workflow.

Over the last few months, that work resulted in a deployment platform built around GitHub Organizations, GitHub Actions, Ansible, k3s, SOPS and an internal application deployment operator written in Go.

This article covers the current version of that platform, the decisions behind it, and some of the problems encountered while building it.

Repository Management

Repositories were moved into a GitHub Organization and branch protection became consistent across projects.

Rulesets were configured to prevent direct pushes to main, require pull request reviews, enforce successful CI checks before merging and restrict who could bypass those requirements.

Since deployments would eventually be tied to merges into main, repository rules became part of the deployment platform itself. A failed CI check blocks a deployment. A missing review blocks a deployment. Direct pushes to main are no longer possible.

The result is that every change reaching production passes through the same validation path regardless of which repository it originated from.

Continuous Integration

The next step was standardizing CI across repositories using GitHub Actions. Most of our services are written in Go, so the workflow was fairly similar:

Run formatting checks
Run linters
Execute tests
Build the application
Build and publish container images

Once CI became consistent across repositories, deployments could assume the same validation process had already taken place regardless of which application was being deployed.

Infrastructure Architecture

Most applications had predictable resource consumption and did not require the elasticity that AWS is particularly good at providing.

A comparison between AWS infrastructure costs and equivalent VPS resources on Contabo made the migration decision fairly straightforward, and several workloads were moved onto VPS infrastructure.

The resulting architecture was built around Contabo's private networking feature. Each server was connected to a private network and cluster communication was configured to use private IP addresses. Kubernetes control plane traffic, pod networking, database connections and application-to-application communication remained on the private network, while public access was limited to services that actually needed internet exposure.

The architecture ended up looking something like this:

Internet
    |
Traefik Ingress
    |
k3s Cluster
    |
+---------------+---------------+---------------+
|               |               |               |
Control Plane   Worker 1      Worker 2      Worker N
     \             |             |             /
      \____________Private Network____________/

Using private networking meant node-to-node communication never needed to traverse the public internet, and new nodes could join the cluster using private addresses rather than public endpoints.

Infrastructure Provisioning

As the number of servers increased, infrastructure provisioning became repetitive.

Every machine required:

User creation
SSH configuration
Hostname configuration
Firewall configuration
Package installation
Cluster bootstrap steps

The first few servers were configured manually. Ansible appeared shortly afterwards.

Most of the provisioning process eventually became playbooks covering server configuration, SSH setup and k3s installation. The value of this became obvious when one of the worker nodes failed.

Replacing the node involved provisioning a replacement server, updating inventory and rerunning the playbooks rather than manually rebuilding configuration.

Deploying k3s

I evaluated kubeadm briefly before choosing k3s.

For the size of infrastructure I was managing, k3s solved most of the problems I cared about without introducing additional operational overhead.

The control plane, service discovery, ingress controller and storage provisioner were already packaged together.

I wasn't particularly interested in assembling Kubernetes components individually if a distribution already existed that solved the same problem.

Within a few hours the cluster was operational and workloads started moving over.

For a small engineering team operating a handful of products, the defaults provided by k3s have been difficult to argue against.

Deployment Automation

Around this period I started getting tired of maintaining Kubernetes manifests.

Most applications required the same collection of resources:

Deployments
Services
Ingresses
ConfigMaps
Secrets

The differences between applications were usually small, but they still resulted in maintaining large amounts of repetitive YAML.

I initially used Helm, but over time I found myself spending more effort maintaining templates and manifests than I wanted to.

Eventually I moved deployment definitions into Go using the Kubernetes client libraries and controller-runtime.

Working directly with the Kubernetes API provided a much deeper understanding of Kubernetes resources than maintaining YAML manifests. Deployments, Services, ConfigMaps and Secrets became resources constructed and managed directly in code.

Writing deployment definitions directly in Go became repetitive over time, so I started building an internal application deployment operator that abstracts much of the Kubernetes resource configuration behind a simpler deployment definition.

The operator is still evolving, but it has become the primary way applications are deployed into the cluster.

I'll cover its design and implementation in a future article.

Secret Management

Keeping environment variables synchronized across applications and environments became increasingly difficult as the platform grew.

I looked at Vault and Infisical, but both introduced another service to operate. For the size of the team and operational requirements, SOPS with Age encryption was a better fit.

Environment files are encrypted with SOPS and committed to Git. During deployment, the files are decrypted and converted into Kubernetes ConfigMaps and Secrets.

The Age private key is stored outside the repository and is only available to trusted deployment environments responsible for decrypting configuration during deployments.

The deployment workflow looks roughly like this:

.env
  |
SOPS + Age Encrypt
  |
Git Repository
  |
Deployment Environment
  |
Age Private Key
  |
Decrypt
  |
Kubernetes Secrets / ConfigMaps

Kubernetes RBAC

Giving GitHub access to deploy applications introduced another problem: authentication to the cluster.

Kubernetes RBAC provides several ways to solve this, typically through service accounts and scoped permissions.

The implementation worked, but managing service account tokens quickly became an operational concern.

At the moment I use a combination of restricted Kubernetes identities and trusted administrative machines for deployment operations.

I am currently evaluating ArgoCD as a GitOps layer on top of the deployment platform, allowing the cluster to reconcile changes from Git rather than accepting deployments pushed directly from GitHub.

Stateful Workloads

Stateful workloads required a different approach from application workloads.

Running databases as StatefulSets works, but using local-path storage introduces an important limitation, the persistent volume is tied to the node where it was created. If that node goes down, Kubernetes can reschedule the pod, but the data does not automatically move with it.

For now, databases still run as StatefulSets on local-path storage, with backups and restore procedures treated as the primary recovery path. This keeps the architecture simple, but it also means node failure recovery depends on either recovering the original node or restoring the database from backup.

Most small teams eventually move toward dedicated database infrastructure, managed database services or distributed storage solutions such as Longhorn as their operational requirements grow.

Current Architecture

Today the platform runs on a k3s cluster hosted on Contabo infrastructure and connected through private networking.

GitHub Actions handles CI, Ansible provisions infrastructure, SOPS manages secrets and applications are deployed through an internal application deployment operator built on top of the Kubernetes API.

Profiling gorilla/websocket fan-out bottlenecks in Go

ude-p — Tue, 19 May 2026 21:20:45 +0000

I have been working on a realtime multiplayer server in Go.

The transport layer is WebSocket using gorilla/websocket, and most outbound payloads are protobuf messages generated with protobuf-go-lite.

The core server loop is simple enough:

accept client input
advance simulation
build world snapshot
broadcast snapshot to connected clients

Each simulation instance runs at 60 ticks per second, so every tick has roughly 16ms available for input processing, simulation, synchronization, and outbound networking.

At small connection counts, everything felt fine. Once I started testing hundreds of clients connected to the same simulation instance, the problems became obvious. Joining became slower, movement updates started lagging behind, snapshots backed up, and tick duration became unstable.

At that point the useful question stopped being:

why is the server slow?

and became:

what part of the realtime path gets worse as connection count grows?

The answer was mostly fan-out.

The server shape

Each WebSocket connection owns a session object.

The session manages:

the websocket connection
outbound queues
heartbeat state
write synchronization
inbound rate limiting

All websocket writes are intentionally serialized.

gorilla/websocket allows:

one concurrent reader
one concurrent writer

In practice, it is still easy to accidentally write from multiple goroutines:

the normal write loop
heartbeat responses
disconnect handlers
internal events

So every socket write goes through one locked path:

func (s *UserSession) write(data []byte) error {
    s.writeMu.Lock()
    defer s.writeMu.Unlock()

    if err := s.Conn.SetWriteDeadline(time.Now().Add(wsWriteTimeout)); err != nil {
        return err
    }

    return s.Conn.WriteMessage(websocket.BinaryMessage, data)
}

That lock was not the main bottleneck, but it matters for correctness. I do not want multiple goroutines calling WriteMessage on the same connection.

The simulation state is owned and mutated by a single goroutine.

Every simulation instance has one command channel handling:

joins
leaves
player input
disconnects
internal events

Each tick roughly looks like this:

drain commands
advance simulation
build synchronization state
broadcast updates

The last step became the expensive one.

The shape that broke first

The simulation work itself was not the first thing to fail.

The first bad shape was the broadcast path.

Each simulation tick builds a snapshot containing replicated world state:

updatedObjects := scratch.updatedObjects[:0]

for objectID, object := range simulation.objects {
    if !object.shouldReplicate {
        continue
    }

    updatedObjects = append(
        updatedObjects,
        syncSnapshotObject(objectID, object),
    )
}

Only objects marked for network replication are included in the snapshot.

That snapshot then gets sent to every connected client subscribed to the simulation instance.

The important detail here is that the snapshot payload is usually identical for every client in that room.

The original broadcast path effectively looked like this:

for _, recipient := range recipients {
    snapshot := buildSnapshot()
    data, _ := snapshot.MarshalVT()

    recipient.Send(data)
}

At small scales, this feels harmless.

At larger scales, the shape becomes expensive very quickly.

With 500 connected clients running at 60 ticks per second:

500 recipients x 60 snapshots/sec
= 30,000 websocket snapshot writes/sec

But the bigger problem was not the socket writes themselves.

The expensive part was repeatedly serializing the same snapshot payload for every connected client.

That means every tick was doing:

protobuf marshaling per recipient
allocations per recipient
buffer growth per recipient

even though the payload itself was identical.

The first profiling lesson was simple:

if the payload is identical for every recipient, serializing it per client is wasted work

Profiling setup

The server uses Pyroscope for continuous profiling:

runtime.SetMutexProfileFraction(5)
runtime.SetBlockProfileRate(5)

pyroscope.Start(pyroscope.Config{
    ApplicationName: "server",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocSpace,
        pyroscope.ProfileInuseSpace,
        pyroscope.ProfileGoroutines,
    },
})

Tick duration is also exported through OpenTelemetry:

tickDuration.Record(
    context.Background(),
    time.Since(tickStartedAt).Seconds(),
)

The metrics I cared about for this issue were:

active websocket sessions
room tick p50/p95/p99
protobuf marshal cost
allocation pressure
websocket backlog growth
goroutine buildup
socket write duration

The profiling run made the problem obvious.

Reliable vs unreliable messages

One important change was separating outbound traffic by semantics.

Not every websocket message deserves the same delivery behavior.

Snapshots are latest-state messages. If the client misses one snapshot, the correct behavior is usually to apply a newer one, not replay old movement history.

Some messages are different:

object created
object deleted
session ended
important gameplay events

Those are ordered state transitions and should not be dropped.

The session ended up with two outbound paths:

Reliable(frame OutboundFrame)
Unreliable(frame OutboundFrame)

Both paths operate on an OutboundFrame.

An OutboundFrame is the object the server uses to represent a websocket payload that is ready to be written. In this case, it carries the marshaled protobuf bytes and the release logic needed when those bytes come from a pool.

Reliable messages use a bounded queue:

Outbound chan OutboundFrame

Snapshots use a latest-only slot:

LatestUnreliable OutboundFrame
UnreliableNotify chan struct{}

The unreliable path intentionally avoids replacing a snapshot that is already waiting to be written:

func (s *UserSession) Unreliable(frame OutboundFrame) {
    s.outboundMu.Lock()

    if s.LatestUnreliable.Data != nil {
        s.outboundMu.Unlock()
        frame.Release()
        return
    }

    s.LatestUnreliable = frame
    s.outboundMu.Unlock()

    select {
    case s.UnreliableNotify <- struct{}{}:
    default:
    }
}

This part is easy to get wrong.

The simulation loop may call Unreliable 60 times per second. If every new snapshot immediately replaced the pending one, a busy session could keep overwriting the pending frame before the writer loop gets a chance to consume it.

So the rule is simple:

if no unreliable snapshot is pending, publish one
if one is already pending, drop the newer one and let the writer catch up

That keeps each session bounded to at most one pending unreliable snapshot.

Reliable messages queue because they represent state transitions. Unreliable snapshots do not queue because they represent latest world state.

A slow client may miss movement snapshots, but it should not force the server to retain stale movement history.

Marshaling once per broadcast

The fix was to move protobuf marshaling out of the recipient loop.

The current broadcast path marshals once:

snapshotFrame := MarshalOutboundFrame(
    snapshotEnvelope,
    snapshotRecipients,
)

Then shares the same frame across sessions:

for _, recipient := range recipients {
    recipient.Unreliable(snapshotFrame)
}

That changes the scaling shape from:

marshal protobuf once per recipient
to:
marshal protobuf once per broadcast

The network work is still multiplied by recipient count.

The serialization work is not.

Sharing broadcast frames safely

Once protobuf marshaling moved out of the recipient loop, the server needed a safe way to share the same websocket payload across many sessions.

The broadcast path now produces one shared OutboundFrame:

type OutboundFrame struct {
    Data    []byte
    refs    *atomic.Int32
    release func()
}

The frame contains:

the marshaled websocket payload
a reference counter
a cleanup callback for returning pooled buffers

The important detail is that the byte slice comes from a pool.

The server marshals protobuf data into a reusable buffer:

buf := outboundFrameBufferPool.Get().(*writeBuffer)

if cap(buf.data) < size {
    buf.data = make([]byte, size)
}

data := buf.data[:size]

_, err := envelope.MarshalToVT(data)

That same byte slice then gets shared across every recipient:

for _, recipient := range recipients {
    recipient.Unreliable(snapshotFrame)
}

At that point ownership matters.

The pooled buffer cannot go back into the pool until every session has either written or dropped the frame.

So the frame carries a reference count:

func (f OutboundFrame) Release() {
    if f.refs == nil {
        return
    }

    if f.refs.Add(-1) == 0 && f.release != nil {
        f.release()
    }
}

Each session releases the frame after writing or dropping it:

err := s.write(frame.Data)
frame.Release()

Without ownership tracking, pooled buffers become dangerous very quickly. One session can still be writing bytes while another goroutine has already returned the buffer to the pool for reuse.

The optimized broadcast path eventually became:

build snapshot once per tick
marshal protobuf once per broadcast
write into pooled buffer
share frame across recipients
release pooled buffer after all recipients finish
drop stale unreliable snapshots

None of these changes remove the cost of socket writes. Every connected client still needs its own websocket write.

What they remove is the avoidable work around the write:

repeated protobuf marshaling
repeated buffer allocation
stale snapshot queue buildup
unnecessary garbage collector pressure

Profiling result

This was the test shape:

simulation tick rate: 60Hz
connected clients: ~500
transport: gorilla/websocket
payload format: protobuf
snapshot delivery: unreliable latest-only

The profiling result looked like this:

Metric	Unoptimized	Optimized
Snapshot builds per tick	~500	1
Snapshot marshals per tick	~500	1
Snapshot marshals/sec	~30,000	~60
Pooled frame buffers	no	yes
Snapshot delivery	queued per recipient	latest-only
Room tick p99	~243ms	~18–22ms
Allocation pressure	very high	much lower
WebSocket backlog growth	severe during bursts	reduced
CPU time in protobuf marshal path	dominant hotspot	mostly removed from broadcast multiplier

The important part is not just the p99 number.

The important part is the shape change.

The optimized version still performs one websocket write per connected client, but it no longer rebuilds and serializes identical snapshot payloads per recipient.
That moved protobuf encoding and buffer allocation out of the recipient loop.

Takeaways

The issue was not that gorilla/websocket is slow.

The issue was that fan-out multiplies small costs very aggressively.

At one client:

protobuf marshaling is noise
allocations are noise
queue pressure is noise

At hundreds of clients and 60 ticks per second, those costs become visible very quickly in:

CPU usage
allocations
websocket backlog growth
tick latency

The main changes were:

separate reliable events from snapshots
make snapshots latest-only
marshal broadcast payloads once
share pooled frames across sessions
use ref-counted ownership for shared buffers

There is still a larger scaling problem left.

Right now every replicated object is still sent to every subscribed client. That means the network shape is still:

objects x recipients

The next step is proper interest management:

visibility filtering
area-of-interest replication
per-client relevance filtering

But I would not start there.

First:

profile the current system
remove multiplied work
fix ownership problems
stabilize the hot path

Then decide whether the architecture actually needs a larger cut.