Architecture Lab

How I design systems

These are trade-offs I have had to make and defend, not a pattern catalogue. Each one is written with what it costs and when not to reach for it, which is the half that usually goes missing.

The shape of a web system

Almost every system I work on is a variation of this. Knowing the default is what makes a deviation from it a decision rather than an accident.

The reference request path: what a request passes through, and which parts of the work happen after the response has already gone back.

Read this diagram as text

UserClient: A browser or a mobile app. Everything downstream exists to answer one of these as quickly as it can.
CDNEdge: Serves static assets and cacheable responses from a location near the user. Every request it absorbs is one your infrastructure never pays for.
API GatewayEdge: One entry point: routing, authentication, rate limiting and request shaping. It is where cross-cutting concerns live so services do not each reimplement them.
ApplicationService: Where the business rules live, whether that is one deployable or fifteen. This is the part worth designing carefully; the rest is plumbing around it.
CacheCache: Holds the results of expensive work so it is done once instead of every request. Fast, and the source of most stale-data bugs.
QueueQueue: Work the user does not need to wait for: email, reports, thumbnails, sync to a third party. It moves latency out of the request instead of removing it.
WorkersService: Consume queued jobs on their own schedule, and scale independently of the traffic that produced them.
DatabaseData: The source of truth, and usually the hardest thing to scale: it is the one component that cannot simply be duplicated without deciding what consistency means.
Third partiesExternal: Payments, identity, messaging, maps. Systems you do not control, cannot fix, and must design around failing.

Connections

User → CDN (static + cacheable)
User → API Gateway (API request)
API Gateway → Application (routed, authenticated)
Application → Cache (read first)
Application → Database (read / write)
Application → Queue (defer work)
Queue → Workers (consume)
Workers → Database (write back)
Workers → Third parties (call out)

The trade-offs

Every one of these buys something and charges for it, and the charge is the part worth reading.

Caching

Do the expensive thing once, then stop doing it.

Caching: Do the expensive thing once, then stop doing it.

Read this diagram as text

ApplicationService: Checks the cache before it touches the database.
DatabaseData: Consulted only on a miss, then the result is written back.
CacheCache: Returns in microseconds when the key is present.

Connections

Application → Cache (1 · read)
Cache → Database (2 · on miss)
Database → Cache (3 · populate)

The problem it solves: The same expensive computation or query runs over and over for results that barely change. Read-heavy systems spend most of their capacity re-deriving answers they already had.
How it works: Keep the result somewhere fast (in memory, in Redis, at the CDN), keyed by whatever the result depends on. Reads check the cache first. Writes either invalidate the key or let it expire.
What it costs: Every cache introduces a window where the world has changed and your answer has not. You are trading correctness-in-time for speed, and you have to decide how much of it you can afford, key by key rather than globally.
When not to reach for it: The data is written as often as it is read, the query is already fast, or the correct answer changes per user in ways the key cannot capture. Caching a slow query is also how a missing index survives to production.

Read the full write-up on Caching

Load balancing

Spread requests across identical instances, and stop sending them to the sick one.

Load balancing: Spread requests across identical instances, and stop sending them to the sick one.

Read this diagram as text

ClientsClient: Address one endpoint and know nothing about what is behind it.
Load balancerEdge: Distributes requests and health-checks every instance.
Instance AService: Identical to B and C. Holds no state that another instance would need.
Instance BService: Can be replaced mid-deployment without a client noticing.
Instance CService: Removed from rotation the moment its health check fails.

Connections

Clients → Load balancer
Load balancer → Instance A
Load balancer → Instance B
Load balancer → Instance C

The problem it solves: One instance is a capacity ceiling and a single point of failure at the same time. You cannot deploy without downtime, and one bad process takes the whole service with it.
How it works: A balancer in front of several identical instances distributes requests and health-checks each one. Failing instances are pulled out of rotation; new ones join it. Deployments become a rolling replacement rather than an outage.
What it costs: Your application must become stateless, which is a real constraint: no in-process sessions, no local file writes, no assuming two requests from one user land in the same place. It also moves the single point of failure to the balancer, so that has to be redundant too.
When not to reach for it: A single instance is comfortably within capacity and the business can tolerate the restart window. Two instances behind a balancer is a meaningful step up in operational complexity. Take that step when you need availability, not because it looks more professional on a diagram.

Read the full write-up on Load balancing

Database replication

Copy the database so reads can go somewhere the writes are not.

Database replication: Copy the database so reads can go somewhere the writes are not.

Read this diagram as text

ApplicationService: Routes writes to the primary and reads to a replica.
PrimaryData: The only node that accepts writes. Streams its change log outward.
ReplicaData: Serves reads, milliseconds behind the primary.
ReplicaData: Also a failover candidate, and where heavy reporting queries belong.

Connections

Application → Primary (writes)
Application → Replica (reads)
Primary → Replica (replicate)
Primary → Replica (replicate)

The problem it solves: Read traffic saturates the one machine that also has to accept every write. Reporting queries slow down checkout. There is also no second copy if the disk dies.
How it works: A primary accepts writes and streams its change log to one or more replicas. Reads are routed to replicas; writes always go to the primary. The same mechanism gives you a failover candidate and a place to run analytics.
What it costs: Replication lag. A replica is behind the primary by milliseconds or seconds, so a user can write something and then not see it: the read-your-own-writes problem, and the source of the "I saved it and it disappeared" bug report.
When not to reach for it: The load is write-heavy, since replicas do not help writes at all. Also when the real problem is a missing index or an N+1 query; replication multiplies a bad query rather than fixing it.

Read the full write-up on Database replication

Queues & async work

Take the slow work out of the request the user is waiting for.

Queues & async work: Take the slow work out of the request the user is waiting for.

Read this diagram as text

APIService: Accepts the request, enqueues the work, responds immediately.
QueueQueue: Durable buffer. Absorbs spikes that would otherwise become timeouts.
WorkersService: Consume and retry independently, scaled to the backlog rather than to traffic.
Dead letterQueue: Where a job goes after it has failed enough times to need a human.
Third partyExternal: The slow or unreliable dependency the user should never wait on.

Connections

API → Queue (enqueue)
Queue → Workers (consume)
Workers → Third party (call)
Workers → Dead letter (after N retries)

The problem it solves: A request that sends three emails, generates a PDF and calls a payment provider is as slow as the sum of those, and it fails if any one of them does.
How it works: The request writes a job to a queue and returns. Workers consume jobs independently, retry on failure with backoff, and scale separately from web traffic. A spike becomes a longer queue rather than a timeout.
What it costs: The system becomes eventually consistent and considerably harder to reason about. You now need idempotent jobs, a retry policy, a dead-letter queue, and a way to tell a user that something they triggered has not finished yet.
When not to reach for it: The user genuinely needs the result before the response; you cannot queue an authorisation check. And a queue relocates a slow operation rather than fixing it. If the work is slow because it is badly written, it will be just as slow in a worker.

Read the full write-up on Queues & async work

Modular monolith vs microservices

Where the boundaries go decides it; the deployable count does not.

Modular monolith vs microservices: Where the boundaries go decides it; the deployable count does not.

Read this diagram as text

Module: OrdersService: Owns its tables. Other modules ask it rather than querying its data.
Module: BillingService: A published interface instead of a shared schema, so it can be extracted later without archaeology.
Module: CatalogueService: The first candidate for extraction if it needs to scale on its own.
DatabaseData: One engine, but schemas kept separate per module: the boundary that makes a later split possible.

Connections

Module: Orders → Module: Billing (explicit interface)
Module: Orders → Module: Catalogue (explicit interface)
Module: Orders → Database (own schema)
Module: Billing → Database (own schema)
Module: Catalogue → Database (own schema)

The problem it solves: Teams block each other in one codebase, deployments become high-risk events, and one component needs to scale differently from the rest.
How it works: A modular monolith enforces boundaries inside one deployable: modules own their data, talk through explicit interfaces, and cannot reach into each other. Microservices take the same boundaries and put a network between them; you gain independent deployment and scaling, and you take on distributed-systems problems in exchange.
What it costs: Microservices convert every in-process call into a network call that can be slow, fail, or arrive twice. You inherit service discovery, distributed tracing, versioned contracts, partial failure and data that no longer joins. Most of the cost is operational, and it lands on the team rather than on the architecture diagram.
When not to reach for it: The boundaries are not yet clear, and in a young product they usually are not. Splitting on the wrong seam produces a distributed monolith: you pay the network cost without getting the independence. Get the modules right in one deployable first; that work is not wasted, because it is exactly the work a split requires.

Read the full write-up on Modular monolith vs microservices

Horizontal scaling

Add more machines instead of a bigger one, once the application allows it.

Horizontal scaling: Add more machines instead of a bigger one, once the application allows it.

Read this diagram as text

Load balancerEdge: The stable address in front of a changing fleet.
App ×NService: Interchangeable instances, added and removed against a load signal.
Session storeCache: Shared state moved out of the process so instances stay disposable.
Object storageInfra: Uploads live here rather than on an instance disk that is about to be terminated.
DatabaseData: The layer that does not scale by duplication, and therefore the one that usually binds first.

Connections

Load balancer → App ×N (distribute)
App ×N → Session store (shared state)
App ×N → Object storage (files)
App ×N → Database (the real ceiling)

The problem it solves: Vertical scaling runs out. There is a largest instance, it is expensive out of proportion to its size, and it is still one machine that can fail.
How it works: Make instances interchangeable (no local state, no local sessions, no local uploads), then run as many as the load needs, adding and removing them automatically against a signal like CPU or queue depth.
What it costs: Statelessness is a design constraint that reaches into the whole application. Sessions move to a shared store, uploads move to object storage, background jobs need locks so two instances do not run the same one, and every scheduled task needs to be leader-elected or idempotent.
When not to reach for it: The bottleneck is the database. Adding application instances to a saturated database makes the problem arrive faster and with more connections. The fix belongs at the constrained layer, however convenient the other one is to scale.

Read the full write-up on Horizontal scaling

CDN & edge delivery

Serve from near the user. The fastest request never reaches you.

CDN & edge delivery: Serve from near the user. The fastest request never reaches you.

Read this diagram as text

UserClient: Resolved to the nearest edge location rather than to the origin.
EdgeEdge: Serves the cached copy. Most requests end here.
OriginService: Consulted on a miss or an expiry, and sets the rules the edge obeys.
Object storageInfra: Where the assets actually live.

Connections

User → Edge (nearest location)
Edge → Origin (on miss only)
Origin → Object storage (fetch)

The problem it solves: Distance is latency you cannot optimise away in code. A user in Stockholm hitting an origin in Cairo pays the round trip on every asset, and your origin pays for traffic that never needed to be dynamic.
How it works: Static assets and cacheable responses are stored at edge locations worldwide and served from the closest one. The origin sets cache headers; the edge does the rest. Images, scripts, fonts and whole pages can be served without a request ever reaching the application.
What it costs: Invalidation. Content is now in hundreds of places, and getting it out of all of them is neither instant nor free. Cache headers become part of the application design rather than a deployment detail, and a wrong `Cache-Control` can pin a broken deploy in front of users for hours.
When not to reach for it: The response is personalised or authenticated, unless you are deliberate about the cache key. A CDN in front of user-specific responses is how one person's data gets served to another, which is the most expensive caching bug there is.

Read the full write-up on CDN & edge delivery

Authentication & authorisation

Who are you, and what are you allowed to do? Two questions, not one.

Authentication & authorisation: Who are you, and what are you allowed to do? Two questions, not one.

Read this diagram as text

Identity providerExternal: Authenticates and issues tokens. May be yours or a federated third party.
ClientClient: Holds a short-lived token and refreshes it, never the password.
GatewayEdge: Verifies the token signature and rejects anything unsigned or expired.
ServiceService: Checks authorisation per action, against the actual record. This is the check that matters.
DataData: Reached only after both questions have been answered.

Connections

Client → Identity provider (1 · authenticate)
Identity provider → Client (2 · token)
Client → Gateway (3 · request + token)
Gateway → Service (4 · verified identity)
Service → Data (5 · authorised access)

The problem it solves: Identity has to be established once and then trusted across every surface (web, mobile, third-party integrations) without each of them reimplementing it, and without a password ever travelling further than it must.
How it works: An identity provider authenticates and issues a short-lived token. Services verify the token rather than calling back for every request. Authorisation is separate and evaluated per action against roles or policies, at the point where the resource is actually touched.
What it costs: Stateless tokens are fast precisely because nobody checks in, which means revoking one before it expires needs a mechanism you have to build. Short expiry plus refresh is the usual answer, and it adds a rotation path that has to work when the network does not.
When not to reach for it: Do not put authorisation only in the gateway or only in the UI. The gateway knows who is asking, but not whether this particular record belongs to them. Hiding a button is not a permission check.

Read the full write-up on Authentication & authorisation

Ask the architect

Common situations, and how I would actually approach them. The order of the investigation is the point: anyone can list the layers of a system, but knowing which one to open first takes years.

These answers are written by hand, not generated. A model improvising architecture advice under someone’s name would be a worse answer and a worse promise.

My API is slow

The symptom

Response times are climbing. It may be all endpoints, or one endpoint that everything else waits behind. Establish which before anything else, because they have different causes.

What I would look at, in order

Where is the time actually going? Get a breakdown per request: database, external calls, application, serialisation. Guessing here wastes weeks.
Database queries first: N+1 patterns, missing indexes, queries that scan instead of seeking. This is the answer more often than everything else combined.
Payload size. A fast query serialising ten thousand rows is not a fast endpoint.
External calls in the request path, and whether they have timeouts. One slow third party becomes your latency.
Contention: connection pool exhaustion, lock waits, a saturated worker pool. These look like "everything is slow" rather than "this is slow".
Only then infrastructure: CPU, memory, network, noisy neighbours.

The approach

Profile, find the single largest contributor, fix that one thing, measure again. Repeat until it is fast enough, then stop. Optimising a layer that accounts for 4% of the time is work you can prove was pointless.

The trap

Adding a cache first. It hides the slow query, makes the problem intermittent instead of constant, and moves the eventual debugging session to a worse day.

My database is overloaded

The symptom

High CPU or I/O, connections piling up, queries queueing. The instinct is a bigger instance; that buys time and tells you nothing.

What I would look at, in order

The slow query log. Rank by total time consumed, not by individual duration: a 30 ms query running 50,000 times a minute outweighs a 3-second report.
Query plans on the top offenders. Sequential scans on large tables are usually a missing or unusable index.
The read/write ratio. It determines whether replicas will help at all.
Connection count against the pool configuration. Thousands of idle connections is an application problem, not a database one.
Write amplification: over-indexing, triggers, and updates that touch far more than they need to.

The approach

Fix the top few queries first; indexes and access patterns typically recover more headroom than any hardware change. Then, if reads dominate, add replicas and route reads to them. Then cache what is genuinely hot and tolerates staleness. Consider sharding last: it is a one-way door.

The trap

Scaling the instance up. It works, briefly, and it converts an engineering problem into a monthly bill that grows with the bug.

My application needs to scale

The symptom

"We need to scale" is not yet a problem statement. Scale to what, on which dimension, by when: traffic, data volume, tenants, or team size? Each has a different answer.

What I would look at, in order

The number. 10× traffic in a year and 10× tomorrow are different projects.
Which dimension is growing: requests, stored data, concurrent users, or write volume.
Where the system breaks first under that dimension. Load-test to find it rather than reasoning about it.
Whether the application is stateless enough to be duplicated at all.
What the business will accept degrading: staleness, eventual consistency, queued work, reduced functionality under peak.

The approach

Find the first ceiling, raise it, then find the next. Systems do not scale uniformly; they hit one constraint at a time, and the second one is often invisible until the first is gone. Design for the next order of magnitude, not for three.

The trap

Rewriting into microservices as a scaling strategy. It addresses team and deployment coupling, not throughput, and it usually makes throughput worse before it makes it better.

Should I use microservices?

The symptom

Usually asked when deployments are risky, teams block each other, or one component needs to scale differently. Those are three different problems and only one of them is solved by splitting.

What I would look at, in order

Are the domain boundaries actually clear? If two teams argue about which service owns a concept, splitting will hard-code that argument into a network call.
What is the real pain: deployment risk, team coupling, or scaling? Deployment risk is often a pipeline and test problem.
Do you have the operational maturity: tracing, centralised logging, CI/CD, on-call, service ownership? Without those, a split multiplies incidents.
How much of the data joins across the proposed boundaries. Frequent cross-boundary joins mean the boundary is in the wrong place.
How large is the team? Under roughly one team per service, you get the costs and none of the independence.

The approach

Enforce the boundaries inside the monolith first: separate modules, separate schemas, no reaching across. That is the same work an extraction would require, so none of it is a detour. Then extract the one component with a genuinely different scaling profile or a clearly separate owner, and see what it costs before extracting a second.

The trap

Splitting by technical layer instead of by domain. Services named after tiers rather than capabilities produce a distributed monolith: every feature touches every service, and nothing can deploy alone.

Should I use Redis?

The symptom

Usually a proxy for "things are slow and caching seems like the answer". Sometimes that is right, but it is worth naming the actual job first.

What I would look at, in order

What exactly is going in it: cache, sessions, queue, rate limiter, locks, leaderboard? Redis is good at all of these and they have different failure modes.
Is the underlying query slow because of a missing index? Fix that first, or you will be caching a bug.
What staleness the data tolerates, expressed in seconds. If the answer is zero, a cache is the wrong tool.
What happens when Redis is unavailable. Does the application degrade or stop?
Whether it needs to survive a restart: cache and job queue want very different durability settings.

The approach

If you need shared state across instances that must be fast and can be lost, Redis is a good answer: sessions, rate limits, hot reads with a defined TTL. Set an explicit expiry on every key from day one, and decide invalidation at write time rather than discovering it in production.

The trap

Treating it as a durable store. Data that only exists in Redis is data you have agreed to lose, whether or not you have realised it yet.

How should I structure my backend?

The symptom

Usually asked while the codebase is still small, which is the right time. Structure is cheap now and expensive in two years.

What I would look at, in order

What the system is about. Structure follows the domain, not the framework tutorial.
What changes together. Code that changes together belongs together, and a folder per technical layer scatters every feature across the tree.
Where the transactional boundaries are: what must succeed or fail as one unit.
What is genuinely shared versus what merely looks similar. Premature sharing produces the coupling that is hardest to undo.
How the team is organised, honestly. Conway's law is not a warning, it is a prediction.

The approach

Organise by domain module (orders, billing, catalogue), each owning its data and exposing an explicit interface. Keep the framework at the edges: HTTP, jobs and CLI are entry points into the domain, not where the domain lives. That gives you testability now and an extraction path later, without committing to distribution before you need it.

The trap

Layer-first folders: controllers, services, repositories, models. Every feature spreads across four directories, nothing owns anything, and the "services" folder becomes the place logic goes to hide.

How can I improve my engineering team?

The symptom

Delivery is unpredictable, quality is inconsistent, or good engineers are leaving. Rarely a talent problem; usually a systems problem.

What I would look at, in order

Where work actually stops. Track a few real tickets end to end; the wait states are the problem, not the coding time.
What "done" means, and whether two engineers would answer the same way.
Review practice: how long a pull request waits, and whether review teaches or only gates.
How much of the system only one person understands. Count the single points of knowledge.
How technical decisions get made, and whether anyone can find out why the last one was made that way.
What the on-call load actually is. Nothing degrades a team faster than being woken up by the same thing twice.

The approach

Start with the constraint instead of everything at once. Usually that means shortening the feedback loop: faster CI, faster review, smaller changes. Then make decisions visible so they can be challenged and reused. Then spread knowledge deliberately through pairing and rotation. Process improvements should be reversible experiments with a date to review them, not permanent policy.

The trap

Adding process to solve a trust or clarity problem. More ceremony on top of an unclear goal makes the same confusion slower and better documented.

How I work with teams See it applied

Let’s talk

Have a product, platform or delivery challenge? Let’s talk about turning it into a structured, scalable solution.

Open to technical leadership, product delivery and senior engineering roles, and available for architecture consulting, technical reviews and mentorship. Engagements run as project-based work, contracts, consulting, freelance engagements, remote collaboration and long-term partnerships.

Based in Cairo, Egypt, working remotely with clients across the MENA region and internationally.

[email protected]All contact details

Also on LinkedIn (opens in a new tab)