Building Komhub Engineering: How We Eliminated DevOps Bottlenecks with a Self-Service Engineering Portal

The Problem

At Komerce, our DevOps team had quietly become a queue.

As our engineering org grew past 50 engineers running dozens of microservices across Kubernetes and two clouds (AWS and GCP), the same five requests kept landing in our Slack channel, every single day:

• "Can I get temporary read access to the production database?"
• "What env var is missing in staging?"
• "Can someone port-forward me into this Redis instance?"
• "Who has the OTP for this vendor login?"

None of these were hard problems. That was the issue. We were spending hours a day on work that had no business requiring a human in the loop — while the actual cost was invisible: every manually created database user was a credential nobody tracked the lifetime of, and every shared kubeconfig was an access boundary nobody could audit later.

We had two options: hire more DevOps engineers to keep up with ticket volume, or remove the need for tickets. We built Komhub Engineering, an internal self-service platform, to do the latter.

---

What Changed

The before/after is the clearest way to show the impact:

The pattern across every row is the same: we replaced a person acting as an access broker with a system that expires by default.

---

System Design

Komhub Engineering is a control plane, not an agent. We deliberately avoided running daemons inside customer infrastructure — instead, the portal (Next.js 15, Prisma, PostgreSQL) talks directly to the same APIs a human operator would: cloud provider SDKs, the Kubernetes API, Vault's HTTP API.

Every write operation — a credential issued, a secret synced, a pod deleted — goes through a single audit path before execution. That path records who did it, when, and posts to a Telegram channel in real time if the target is production. We didn't build this as an afterthought; the audit log was a harder requirement than the self-service UX itself, because the entire premise of removing humans from the approval loop rests on being able to prove, after the fact, exactly what happened.

---

The JIT Database Credential Engine

This is the feature that mattered most, because static database credentials were our single biggest standing risk.

The flow is intentionally boring:

1. The portal already knows which databases exist — it discovers them from registered, AES-256-encrypted connection strings, and tags each one by environment (production, staging, development).
2. A request is checked against a permission flag and rate-limited to one credential per database per 5 minutes, so this can't be turned into a way to spray credentials.
3. A user is created with a predictable, traceable name — <email_prefix>_<unix_timestamp> — with a generated password and scoped read/write grants.
4. TTL is not configurable by the requester. Production gets 2 days. Everything else gets 7. This is a deliberate constraint, not a default — we do not want engineers negotiating longer-lived production access through the UI.
5. A cron job sweeps for expired records and runs REVOKE / DROP USER on the target database. If this job fails silently, credentials outlive their TTL — so it's one of the few jobs in the platform that pages on failure rather than just logging.

CREATE ROLE frengky_1718000000 WITH LOGIN PASSWORD 'Secr3t_p@ssw0rd!';
GRANT CONNECT ON DATABASE "ecommerce_prod" TO frengky_1718000000;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO frengky_1718000000;

The result we cared about wasn't speed, though speed was a nice side effect (credential issuance went from a multi-hour ticket to under 5 seconds). The result we cared about was that standing production database credentials went to zero, because there was no longer a reason for anyone to ask DevOps to create one manually.

---

Secrets: Making Drift Visible

Vault KV v2 sits behind the portal for secret management, but the actual win wasn't storing secrets centrally — teams had that before, and still copy-pasted .env files anyway because comparing environments manually was tedious.

What changed adoption was a side-by-side diff view: staging vs. production, rendered so a missing key is visually obvious before a deploy, not after a crash loop in production. Sensitive values stay masked unless the viewer has elevated permissions. Syncing a changed value pushes it into the corresponding Kubernetes Secret and triggers a rolling restart — no manual kubectl apply, no forgotten redeploy step.

---

Kubernetes Access Without Shipping Kubeconfigs

Handing out cluster access used to mean one of two bad options: share an admin kubeconfig (too much access, no expiry), or have DevOps manually open a tunnel (doesn't scale).

We replaced both with short-lived ServiceAccount tokens scoped to pods/portforward only, requested through a TokenRequest, valid for 24 hours. The portal packages a bootstrap script and cluster CA cert, uploads it to S3, and hands the developer a single command:

curl --silent https://s3.ap-southeast-1.amazonaws.com/komerce-devops/port-forward-scripts/frengky-171800.sh | bash

That's the entire interface. No cluster-admin credentials ever touch a developer's laptop, and the token is worthless in 24 hours even if it leaks.

---

What We Didn't Get Right the First Time

Worth being honest about: our first version of the TTL policy set production credentials to expire in 24 hours, not 2 days. In practice, this meant engineers debugging a multi-day incident kept re-requesting credentials mid-investigation, which defeated the point — they started requesting them earlier "just in case," which is the exact hoarding behavior we were trying to eliminate. We moved to 2 days after watching request patterns for a few weeks. The lesson: an expiry policy that's technically more secure but practically annoying gets worked around, and the workaround is usually worse than a slightly longer TTL.

We also underestimated the Gmail API integration's edge cases — domain-wide delegation scopes needed several rounds of tightening after our first pass exposed more of the shared inbox than the OTP-extraction feature actually needed.

---

Results

• Database access latency: from 2+ hours (ticket-based) to under 5 seconds (self-service).
• Standing production database credentials: zero, down from an untracked but non-trivial number.
• 50+ engineers operating independently on port-forwarding, secret diffing, and rollout restarts — no DevOps intervention required for any of the workflows in the table above.
• Every credential, secret sync, and infrastructure action is now attributable to a specific person and timestamp, which we didn't have before at all.

None of these numbers were the goal on their own — they're a byproduct of removing a human from a path that didn't need one.

---

Takeaways

Make the secure path the fast path. Engineers didn't stop using static credentials because we told them to. They stopped because requesting an ephemeral one was faster than asking a person for a permanent one.

Expiry is a better security control than policy. We spent far less time enforcing rotation and revocation once every credential simply stopped working on its own. A rule that requires nobody to remember it is the only rule that reliably holds.

This freed DevOps to stop being a queue. The team's time shifted from provisioning access to building the systems that make provisioning unnecessary — which is a better use of a DevOps engineer, and it's why we'd make the same call again.