Tiny URL

Introduction

Writing an API project in Golang can be likened to art, due to the absence of a de facto framework or established standards for development. There is a plethora of approaches available, and it isn't uncommon to discover that the chosen method isn't as effective or extensible as initially assumed.

Objectives

In this project, I aim to explore and demonstrate the outcomes of several such approaches:

• Logging with zap: A fast, structured, leveled logging in Go.
• Metrics with otel (OpenTelemetry): Instrumenting code to collect and report metrics.
• Tracing with otel (OpenTelemetry): Capturing the flow and latency of operations in our application.
• Dependency Injection using fx: A framework for dependency injection providing a robust way of managing dependencies.
• Migrations using goose: Managing database schema migrations with embedded SQL files.

Packaging

I am following the rules defined by golang-standard. The internal/domain package contains the domain-specific logics. As rule of thumbs everything defined in internal/domain must use only go standard packages or other application packages, so they should not use any third party libraries directly.

The infrastructure layer does the actual using of third party libraries and resides in infra package. Actual implementation always goes into the infra package.

Repositories

To facilitate database access, we've defined repository interfaces within the domain package. These form the contract for our data access methods. Corresponding implementations can be found in the infra package, ensuring a separation of concerns between our domain definitions and infrastructure-specific code.

Repository Interfaces

• The interfaces are prefixed with repo to denote their role as repositories within the domain layer.

Implementation

• The actual implementations carry a db prefix, indicating their direct interaction with the database and their role within the infrastructure layer.

Key Generation

Handing out a short key that nobody else holds is the one problem a url shortener cannot avoid, and there is more than one way to solve it. The generator package implements four, behind a single interface, so they can be compared by changing generator.type in the configuration.

The random generators make no promise of their own. They draw a key, the primary key on urls rejects it if it is taken, and the service draws another — up to urlsvc.MaxKeyGenAttempts times. This holds up because the key space stays sparse: with N keys stored, a fresh one collides with probability N/Space, and the expected number of attempts is 1/(1-load factor). simple draws from math/rand and secure from crypto/rand; only the latter produces keys a stranger cannot guess.

The counting generators never collide at all. counter encodes a Postgres sequence in base62, so distinct identifiers give distinct keys and there is nothing to check and nothing to retry. Its weakness is that consecutive keys are adjacent, so holding one lets anybody walk the rest.

feistel fixes that without giving up the guarantee. It pushes the identifier through a keyed Feistel network before encoding it. A Feistel network is a bijection whatever its round function computes, so distinct identifiers still give distinct keys, while consecutive ones land far apart. It requires generator.key to be set; that secret is what makes the permutation unguessable, and it deliberately has no default.

The sequence behind both counting generators is declared with a cache, so each database session claims a block of identifiers up front and spends none of them on coordination — the same idea as a hand written Hi/Lo allocator. The cost is gaps in the numbering, which cost nothing but key space.

One Key Space

Names chosen by a caller and keys produced by a generator live in the same key space, stored exactly as the caller will type them. Nothing keeps the two apart because nothing needs to: the primary key rejects a name that is taken, and a generated key that happens to land on a claimed name is simply regenerated.

Chosen names used to be stored behind a static_ prefix instead. That guaranteed separation, but it meant the key that was stored was not the key anybody typed, so every lookup of a chosen name spent one query failing to find the typed key before trying the prefixed one. Uniqueness was already guaranteed elsewhere, so the prefix bought a second query and nothing else.

The Read Path

Generating keys is the interesting problem; serving redirects is the one that decides whether the service stays up. A shortener reads far more than it writes, and the naive implementation of a redirect manages to be both a read and a write: it looks the url up, then increments a counter on the row it just found. The busiest links therefore contend on their own row, and every read costs a write.

Two decorators around the repository fix that, configured under repository. Both implement the same interface as the layer they wrap, so either can be switched off and nothing else changes.

• repository.Cached answers lookups from an in-process LRU. What a redirect needs — the target of a key — never changes once written, so the only real question is when to stop trusting an entry. Entries lapse at ttl, or at the url's own expiry if that comes first, so a cached url can never outlive the point where the database would have stopped returning it. Misses are remembered too, for the much shorter negative_ttl, because keys this short invite being guessed at; creating a key clears any miss recorded for it.
• repository.BufferedVisits counts visits in memory and folds them into one statement every flush_interval, or sooner once max_buffered keys have piled up. A shutdown drains what is left. This trades an exact, instantly visible counter — which nothing reads — for one write per batch instead of one per redirect.

Measured on the development setup, 200 redirects of the same key produce 2 SELECTs and no writes at all while they are being served, then a single batched UPDATE carrying all 200 counts.

The caches are per instance, which is the honest limitation of both: a negative entry keeps one instance from seeing a key another instance just created until it lapses, and a crash loses at most one flush interval of counts. Neither is free to fix, and neither is worth fixing with a shared cache until there is more than one instance to disagree.