The simple version works in staging. Here’s what breaks when fifty thousand people click your link in the first ten minutes.
Read here for FREE.
The first version of a URL shortener is a weekend project. A hash function, a database row, a redirect. Forty lines of code if you’re being verbose. Ships in an afternoon, works fine, gets forgotten.
The second version is the one that keeps you up at night.
Not because the code is wrong. It’s not. The code does exactly what it was written to do: look up a short code, return a destination, send the redirect. The problem is that “look up a short code” stops meaning “one database query” the moment your link ends up on the front page of Reddit. It starts meaning twelve hundred concurrent database queries against a table that was never indexed for this access pattern, at the exact second your cache TTL expires.
That’s the failure mode the tutorials skip. This article covers what actually breaks and how to build around it.
What You’re Actually Building
Functional requirements first. A URL shortener does three things:
- Accept a long URL and return a short code
- Redirect anyone who visits the short code to the destination
- Optionally, track how many times each code was used
Non-functional requirements are where it gets interesting. The redirect has to be fast: under 10ms for a cache hit, under 50ms for a cache miss. It has to be available even if the primary database is down. And it has to handle a 10,000x traffic spike without degrading into a cascade of timeouts.
Scale Estimation: What You’re Sizing For
Working with concrete numbers matters in system design because the right answer at 1,000 requests per day is wrong at 10 million.
Start with URLs: assume 100 million total stored. That is a reasonable design target for a system serving a mid-size product or marketing team.
Read-to-write ratio: redirects heavily outnumber URL creations. A reasonable assumption is 100:1. Every new URL gets clicked many more times than it gets created.
Peak redirect load: assume one link goes viral and drives 50,000 simultaneous users to your redirect endpoint. That is your design load. If your system handles 50,000 concurrent redirects without falling over, everything else is easy.
Storage: each URL record needs the short code, the destination URL (2KB max), timestamps and a click counter. At 100 million URLs, that’s roughly 200GB of raw data, well within what a single PostgreSQL instance handles.
The Encoding Problem: How You Turn IDs into Codes
Every URL shortener needs a way to map short alphanumeric codes to database records. There are three approaches, each with trade-offs.
Hash-based encoding
Take the destination URL, run it through MD5 or SHA-256, truncate the output to 7 characters. Simple. The same URL always produces the same code.
The problem: collisions. Two different URLs can produce the same 7-character prefix. You need a collision check (query the DB before inserting) and a fallback. At scale, you’re doing two database round-trips for every new URL.
Random Base62 encoding
Generate a random 7-character string from the Base62 alphabet (digits 0–9, uppercase A-Z, lowercase a-z) and check if it’s taken. No collision from encoding the URL, but you still need a uniqueness check before every insert.
Counter-based Base62 encoding
Maintain a monotonic counter. Each new URL gets the next integer ID. Encode that integer to Base62 for the short code. No collisions by construction. No uniqueness check needed.
Here’s the encoder:
# Base62 encode: convert an integer ID to a short alphanumeric code
CHARSET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
BASE = len(CHARSET) # 62
def encode(n: int) -> str:
if n == 0:
return CHARSET[0]
result = []
while n:
result.append(CHARSET[n % BASE])
n //= BASE
return "".join(reversed(result))
def decode(s: str) -> int:
result = 0
for char in s:
result = result * BASE + CHARSET.index(char)
return resultSeven characters gives you 62⁷, approximately 3.5 trillion unique codes. At 1,000 URL creations per second, you won’t exhaust the space for over 111 years.
Counter-based encoding is the fastest approach. It has no uniqueness check, no collision retry, and the encoding itself is a handful of integer divisions.
It has one problem: sequential IDs reveal your creation rate. If I shorten two URLs an hour apart and receive codes that decode to integers 1,000 apart, I know your service created 1,000 URLs in that hour. Competitors can watch your growth by creating URLs and decoding the codes.
The fix is to reserve ID ranges instead of allocating one at a time. Each application node fetches a block of IDs from the database on startup (say, 1,000 at a time), holds them in memory, and issues them locally without talking to the database for each URL. Between nodes, the blocks are non-overlapping, so there are no collisions. Within each block, IDs are sequential, but the blocks are interleaved across nodes, making the global sequence non-monotonic and much harder to predict.
Trade-off summary
Counter-based:
- ID generation: no DB round-trip per URL (with range reservation)
- Collision check: not needed
- Business metric leakage: yes, unless range reservation is used
- Best for: high-volume URL shorteners where write throughput matters
Random Base62:
- ID generation: requires uniqueness check against DB
- Collision check: required
- Business metric leakage: no
- Best for: low-to-medium volume, or when code unpredictability matters
The Read Path: The Redirect Is Where Everything Breaks
URL creation happens once. The redirect happens thousands of times for every popular link. This is where your system either holds or fails.
The naive version does one thing per redirect: query the database, return a 302. This works fine at low traffic. It fails at high traffic because PostgreSQL is not designed to serve thousands of identical read queries per second when a simpler system (Redis) can answer the same question in under a millisecond.
The correct design is cache-first:
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
def get_destination(short_code: str) -> str | None:
# L1: Redis cache (sub-millisecond reads)
cached = r.get(f"url:{short_code}")
if cached:
return cached
# L2: Database (only on cache miss)
destination = db.fetch_one(
"SELECT destination FROM urls WHERE code = %s", (short_code,)
)
if destination:
r.setex(f"url:{short_code}", 86400, destination) # cache for 24 hours
return destinationThis is the cache-aside pattern. Redis serves the overwhelming majority of redirects. The database is only involved on first access or after a cache expiry.
At a 95% cache hit rate, a reasonable target for a URL shortener where popular links are requested repeatedly, only 5% of redirects touch the database. Your PostgreSQL instance can breathe.
The Thundering Herd: What Kills Your System When a Link Goes Viral
Here is the failure mode that trips up teams that thought they had it figured out.
A link goes viral. It’s being clicked 10,000 times per minute. Your cache is serving every redirect in under a millisecond. Everything looks fine.
Then the TTL expires. In the same fraction of a second, several hundred concurrent requests check the Redis cache, get a miss, and all race to query the database for the destination URL. Each one queries the same row. Each one writes the result back to Redis. Your database goes from handling a handful of cache-miss queries to handling hundreds of simultaneous identical queries.
This is a cache stampede. It’s not a theoretical concern. It’s the most common failure mode for high-traffic URL shorteners, and it gets worse the more popular your links are.
The fix is a distributed lock. Only one request should be allowed to query the database on a cache miss. The rest should wait, then read the result from the cache once the first request has populated it.
import uuid
import time
def get_destination_safe(short_code: str) -> str | None:
# Fast path: cache hit
cached = r.get(f"url:{short_code}")
if cached:
return cached
# Slow path: acquire a lock before hitting the database
lock_key = f"lock:{short_code}"
lock_token = str(uuid.uuid4())
# SET NX EX: atomic "set only if not exists, expire in 5 seconds"
lock_acquired = r.set(lock_key, lock_token, nx=True, ex=5)
if lock_acquired:
try:
destination = db.fetch_one(
"SELECT destination FROM urls WHERE code = %s", (short_code,)
)
if destination:
r.setex(f"url:{short_code}", 86400, destination)
return destination
finally:
# Safe release: only delete if we still own the lock
script = """
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
end
return 0
"""
r.eval(script, 1, lock_key, lock_token)
else:
# Another request is loading the destination; wait briefly and retry.
# This is a simplified single-retry. Production code should use a
# short retry loop with a deadline, or a blocking BLPOP wait on a
# notification key, to handle DB queries that take longer than 50ms.
time.sleep(0.05)
return r.get(f"url:{short_code}")The nx=True flag makes the Redis SET atomic. Only one caller acquires the lock. The ex=5 flag ensures the lock auto-releases if the holder crashes. The Lua script for release prevents accidentally deleting another client's lock if your TTL expired before you finished.
This pattern cuts your database from “hundreds of simultaneous queries for the same row” to “one query per cache expiry, regardless of concurrent traffic.”
The 301/302 Decision: A Business Choice Wearing a Technical Disguise
Most teams pick 302 redirects without thinking about it. Some pick 301 for “performance” without understanding what they’re giving up.
The difference is caching behavior.
A 301 redirect is permanent. Without explicit cache-control headers, browsers cache it indefinitely. After the first visit, the browser redirects directly from its local cache and never contacts your server again for that short code. This means faster redirects for repeat visitors and lower server load. (You can override the caching behavior with Cache-Control: no-store on the response, but most URL shorteners never set that header, so indefinite browser caching is what you actually get in production.)
It also means you have permanently lost the ability to track those clicks. Once the 301 is cached, the request never reaches your server. Your analytics database never sees it. If the link goes viral and half the clicks come from browsers that cached the 301 on their first visit, you are counting half your traffic. You may also never be able to change the destination URL for that code. Browsers will redirect to the cached destination until the cache is explicitly cleared.
A 302 redirect is temporary. Browsers do not cache it. Every click passes through your redirect server, where you can count the request, log the user agent, change the destination on the fly, and run the redirect pipeline against a feature flag or geo-routing rule.
Use 301 when:
- The destination will never change
- You do not need per-click analytics
- Your primary concern is reducing load on the redirect server for repeat visitors
- The link is internal (canonical URL management, permanent resource moves)
Use 302 when:
- You need to track clicks
- The destination might change (A/B test, campaign rotation)
- You need to update where the link points without changing the short code
- The link is for a marketing campaign or referral tracking
Most URL shortener use cases need 302. The one exception is canonical URL redirection in your own infrastructure, where permanence is the point.
If you shipped 301 redirects to production and now want analytics, you cannot recover the cached clicks. The only fix is rotating the short codes: creating new codes for the same destinations so browsers have no cached entry to fall back on.
Analytics Without Blocking the Redirect
The redirect has one job: be fast. Writing a click event to a database table on every redirect turns a sub-millisecond operation into a 20–30ms synchronous write. At high traffic, this creates back-pressure that degrades the entire redirect pipeline.
The solution is fire-and-forget. Log the click asynchronously, never in the redirect path.
from queue import Queue, Full
import time
# In-memory analytics buffer (per-process). With Gunicorn or uWSGI multi-worker
# deployments, each worker process has its own queue. In production, replace
# this with an external queue (SQS, Redis stream) so all workers share one sink.
analytics_queue: Queue = Queue(maxsize=50_000)
def log_click_async(short_code: str, user_agent: str, ip: str) -> None:
"""Queue a click event. Never raises; drops the event on overflow."""
try:
analytics_queue.put_nowait({
"code": short_code,
"ua": user_agent,
"ip": ip,
"ts": time.time(),
})
except Full:
# Under extreme load, drop the event rather than slow the redirect.
# A dropped click is better than a failed redirect.
pass
def redirect_handler(short_code: str, request) -> dict:
destination = get_destination_safe(short_code)
if not destination:
return {"status": 404}
# Fire-and-forget: does not block the redirect
log_click_async(
short_code,
request.headers.get("User-Agent", ""),
request.remote_addr
)
return {"status": 302, "location": destination}A background worker drains the queue and writes to your analytics store in batches. In production, you would replace the in-memory queue with a proper message queue (SQS, RabbitMQ or Kafka) so click events survive application restarts and can be consumed by multiple downstream services.
The key constraint is: the redirect must never wait for the analytics write. Drops under extreme load are acceptable. Failed redirects are not.
Abuse Prevention: Three Problems You Will See
URL creation rate limiting
Without rate limiting on the shortening endpoint, one bad actor can exhaust your ID space or your storage budget. A per-IP rate limit using Redis counters (increment on every request, expire after a window) keeps this manageable. Block at the edge (load balancer or API gateway) before the request reaches application code.
SSRF via destination URLs
A URL shortener that accepts any destination URL becomes an SSRF vector the moment you add any feature that makes server-side requests to that URL: validation (checking if the URL resolves), link preview (fetching Open Graph tags), thumbnail generation, or malware scanning. If your server fetches http://169.254.169.254/latest/meta-data/ during any of those steps, you have handed an attacker your AWS instance credentials. The HTTP 302 redirect itself is safe: your server sends a Location header and the client follows it. The risk is every other feature you build on top.
Validate destination URLs before storing them. The validation must resolve the hostname to an IP address and check that IP against a blocklist of private and link-local ranges: RFC 1918 (10.x, 172.16.x through 172.31.x, 192.168.x), loopback (127.x), link-local (169.254.x) and IPv6 equivalents. Checking only the hostname string is not enough; a hostname can resolve to a private IP while appearing legitimate.
Short code enumeration
Sequential codes (0000001, 0000002…) are trivially enumerable. If you are using counter-based encoding without ID shuffling, an attacker can iterate through all your short codes and scrape every destination URL you have ever stored. This is especially bad for internal links shortened for convenience.
Options: add a minimum code length that makes brute force impractical, require authentication for sensitive URLs, or use random code generation for links that should not be discoverable.
The Write Path at Scale
For most URL shorteners, writes are not the bottleneck. Reads are. But once you are handling tens of thousands of URL creations per day, a few patterns matter.
Write to the primary database only. Reads can go to replicas, but URL creation must hit the primary to guarantee the record exists before you return the short code to the caller.
Use a database-level unique constraint on the short code column, even with counter-based generation. It is a last-line defense against race conditions, not your primary uniqueness strategy.
Index on the short code column, not the destination URL. Every redirect lookup hits this index. Every analytics query hits it. It needs to be fast.
Consider separating the URL metadata table (code, destination, created_at, creator) from the click counter table. Writing a click counter increment on every redirect serializes updates on the same row. Moving click counting to an append-only events table, or batching updates from the analytics queue, avoids that lock contention.
What Makes This System Actually Hard
Building a URL shortener that works is a weekend. Building one that handles a viral moment without degrading (fast redirects, accurate analytics, no thundering herd against the database) takes thought about caching strategy, redirect semantics and the failure modes that only show up at scale.
The 301/302 decision is made once and is difficult to reverse. The thundering herd is invisible until it kills your database in production. Sequential IDs are a business intelligence leak most teams never notice until a competitor mentions your monthly growth number.
None of these are bugs. They are consequences of design choices that look identical in staging and very different in production.
If you are running a URL shortener at scale and redirect latency is the thing keeping you up at night, the distributed lock pattern and async analytics queue are the two changes that move the needle. Everything else is optimization.
