Site Topics Fundamentals 5 in Practice: Lessons From Real Deployments

Backup Strategy: Periodic jobs should be safe to run twice, because they will be. Backup Strategy: You rarely need a new component to fix a boundary problem. Backup Strategy: The signal you want is often already logged, just not aggregated.

Content Delivery: The first thing to settle is the failure mode, not the happy path. Measurements taken once are anecdotes; you need a baseline that repeats. That applies to content delivery as well. In practice, content delivery behaves differently: Costs usually concentrate in a small number of operations, so find those first.

Schema Migration: Periodic jobs should be safe to run twice, because they will be. Schema Migration: You rarely need a new component to fix a boundary problem. Schema Migration: The signal you want is often already logged, just not aggregated.

Teams working on monitoring alerts usually discover this the hard way. If the rollback plan needs a meeting, it is not a rollback plan. Small pages that stay small are easier to keep fast than large ones made fast. This is most visible in monitoring alerts. Consider monitoring alerts specifically. Write the invariant down; otherwise it lives only in someone's memory.

Rate Limiting: The interesting number is not the average, it is the 99th percentile. Rate Limiting: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Rate Limiting: Every abstraction you add is a place where behaviour can differ from intent.

If the conversation becomes tense, you can pause it and return later if you feel safe doing so. You might say, “I’m not continuing this discussion while I’m being pressured,” then leave or contact someone you trust. If you fear retaliation or feel unsafe, consider speaking with a local sexual-violence support service or another qualified professional before confronting the person. Available services and legal protections vary by location.

Log Analysis: Configurations should be reviewable in a diff, not only in a console. Log Analysis: The best time to add an index is before the table gets large. Log Analysis: Failures are usually correlated, so plan for the shared dependency.

Content Delivery: Serving static bytes is the cheapest thing you can do at the edge. Content Delivery: A schema is an interface; changing it is a migration, not an edit. Content Delivery: Track the denominator as carefully as the numerator.

Access Control: If the rollback plan needs a meeting, it is not a rollback plan. Access Control: Small pages that stay small are easier to keep fast than large ones made fast. Access Control: Write the invariant down; otherwise it lives only in someone's memory.

A queue smooths spikes but also hides how far behind you are. This is most visible in api design. Consider api design specifically. Retries without jitter turn a small outage into a large one. API Design: Separating the reads from the writes buys room to change either side.

Observability: If a metric has no owner, it will drift until it causes an incident. Observability: The cheapest optimisation is usually removing work nobody asked for. Observability: Aggregating at write time trades flexibility for predictable read cost.

Storage Tiers: If the rollback plan needs a meeting, it is not a rollback plan. Storage Tiers: Small pages that stay small are easier to keep fast than large ones made fast. Storage Tiers: Write the invariant down; otherwise it lives only in someone's memory.

In practice, data pipelines behaves differently: A queue smooths spikes but also hides how far behind you are. Retries without jitter turn a small outage into a large one. The same reasoning holds for data pipelines. For data pipelines, the constraint matters more than the feature list. Separating the reads from the writes buys room to change either side.

Monitoring Alerts: The interesting number is not the average, it is the 99th percentile. Monitoring Alerts: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Monitoring Alerts: Every abstraction you add is a place where behaviour can differ from intent.

Storage Tiers: You can often replace a coordination problem with an idempotency key. Storage Tiers: Anything that grows without a bound will eventually hit one. Storage Tiers: Documentation that is not tested tends to describe the previous version.

Consider search indexing specifically. If the rollback plan needs a meeting, it is not a rollback plan. Search Indexing: Small pages that stay small are easier to keep fast than large ones made fast. Write the invariant down; otherwise it lives only in someone's memory. That applies to search indexing as well.

In practice, monitoring alerts behaves differently: Configurations should be reviewable in a diff, not only in a console. The best time to add an index is before the table gets large. The same reasoning holds for monitoring alerts. For monitoring alerts, the constraint matters more than the feature list. Failures are usually correlated, so plan for the shared dependency.

You can often replace a coordination problem with an idempotency key. The same reasoning holds for release process. For release process, the constraint matters more than the feature list. Anything that grows without a bound will eventually hit one. Teams working on release process usually discover this the hard way. Documentation that is not tested tends to describe the previous version.

Monitoring Alerts: The first thing to settle is the failure mode, not the happy path. Measurements taken once are anecdotes; you need a baseline that repeats. That applies to monitoring alerts as well. In practice, monitoring alerts behaves differently: Costs usually concentrate in a small number of operations, so find those first.

A design that cannot be rolled back is a design that cannot be changed safely. The same reasoning holds for cloud infrastructure. For cloud infrastructure, the constraint matters more than the feature list. Latency budgets are easier to defend when every hop has a stated ceiling. Teams working on cloud infrastructure usually discover this the hard way. Caching helps only until the invalidation rules become the bottleneck.

Queue Design: The interesting number is not the average, it is the 99th percentile. Queue Design: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Queue Design: Every abstraction you add is a place where behaviour can differ from intent.

Crawl Budget: You can often replace a coordination problem with an idempotency key. Crawl Budget: Anything that grows without a bound will eventually hit one. Crawl Budget: Documentation that is not tested tends to describe the previous version.

Release Process: The first thing to settle is the failure mode, not the happy path. Release Process: Measurements taken once are anecdotes; you need a baseline that repeats. Release Process: Costs usually concentrate in a small number of operations, so find those first.

Edge Caching: A queue smooths spikes but also hides how far behind you are. Retries without jitter turn a small outage into a large one. That applies to edge caching as well. In practice, edge caching behaves differently: Separating the reads from the writes buys room to change either side.