How I Would Walk a Staff Interview on Event-Driven Architecture

A lot of event-driven interview answers sound polished but disconnected. They name Kafka, say “eventual consistency,” mention DLQs, and stop there. That usually falls apart once the interviewer asks what actually happens when an order is placed, a payment is slow, or a poison message hits the system. I wanted this post to stay grounded in one concrete flow from start to finish.

The PRD in this repo uses an e-commerce order fulfillment system, which is a good interview problem because it forces the right tradeoffs. We need low-latency order intake, independent payment and inventory processing, resilience during spikes, and a way to explain why distributed systems stop behaving like a single transaction.

The interview question I would answer

If I translate the PRD into the kind of prompt I have actually heard in interviews, it comes out something like this:

Design an order processing system for an e-commerce platform.
Orders should be accepted quickly, payment and inventory should be handled
asynchronously, and the system should tolerate failures, spikes in traffic,
and malformed messages. Walk through the architecture and the tradeoffs.

That prompt matters because it tells me the interviewer is not asking for a CRUD app. They are testing whether I can connect business goals to architecture decisions and failure handling.

Step 1: Start with the problem, not the tooling

I would start by saying the business requirement is fast order intake, not immediate end-to-end completion. That one sentence drives the rest of the design. If the goal were “complete payment, reserve stock, update analytics, and respond only after all of that succeeds,” then synchronous request chaining might still be on the table. But that is not the requirement here.

That is the first decision: I am intentionally choosing an asynchronous architecture because the problem calls for decoupling and elasticity more than immediate consistency.

Step 2: Define the service boundaries

From there I would carve the system into domains that already want separate lifecycles: Order, Payment, Inventory, and a Read Model or Analytics side. I would avoid creating ten tiny services in the interview. Four clear boundaries are enough to explain the pattern without turning the answer into a service catalog.

flowchart LR
    Client[Client]
    Order[Order Service]
    Broker[Kafka Topics]
    Payment[Payment Service]
    Inventory[Inventory Service]
    Read[Read Model]

    Client --> Order
    Order --> Broker
    Broker --> Payment
    Broker --> Inventory
    Broker --> Read
    Payment --> Broker
    Inventory --> Broker
                    

The reason I like these boundaries is that they map to ownership and failure isolation. Payment can fail because Stripe is slow. Inventory can fail because a warehouse database is degraded. Analytics can lag because a projection job is rebuilding. None of those should stop the platform from accepting a new order.

Step 3: Walk the happy path from request to confirmation

This is the part I would say out loud in the interview, step by step, because each decision should feed the next one.

sequenceDiagram
    participant Client
    participant Order as Order Service
    participant OrderDB as Order DB
    participant Kafka as Kafka
    participant Payment as Payment Service
    participant Inventory as Inventory Service
    participant Read as Read Model

    Client->>Order: POST /api/orders
    Order->>OrderDB: Save order as PENDING
    Order->>Kafka: Publish OrderCreated
    Order-->>Client: 202 Accepted + orderId
    Kafka-->>Payment: OrderCreated
    Kafka-->>Inventory: OrderCreated
    Payment->>Kafka: PaymentSucceeded or PaymentFailed
    Inventory->>Kafka: InventoryReserved or InventoryRejected
    Kafka-->>Order: Payment and inventory events
    Order->>OrderDB: Update status to CONFIRMED or CANCELLED
    Kafka-->>Read: Project events into read model
                

My explanation would go like this:

The client sends POST /api/orders to the Order Service. The Order Service validates the request, stores a new order with a PENDING state, publishes an OrderCreated event to Kafka, and returns 202 Accepted. Payment and Inventory consume that event independently. They do their work and publish the outcome as new facts. The Order Service listens to those downstream events and updates its state. In parallel, the read side builds denormalized views for dashboards and reporting.

That explanation matters because it shows the interviewer I understand the difference between command handling, event propagation, and state convergence.

Step 4: Explain why I return 202 instead of waiting

This is usually where the interviewer pushes: why not just call Payment synchronously from Order?

My answer is that synchronous chaining creates the exact failure shape I want to avoid. If Order calls Payment, Payment calls Inventory, and any one of them is slow, the entire request path inherits that latency. If Payment is down, Order is effectively down. That is fine for a tightly coupled transactional workflow inside one boundary. It is a poor fit for a cross-domain system that needs elasticity.

Step 5: Use Kafka as a buffer, not just a transport

Once I choose asynchronous flow, the next question is why Kafka. In this design Kafka is not just moving messages around. It is absorbing rate mismatch between producers and consumers.

If the Order Service can publish 10,000 OrderCreated events per second during a spike, and the Payment Service can only process 500 per second because it depends on an external gateway, Kafka gives me a buffer. The requests are accepted, the events are durably stored, and the consumer works through the backlog at a controlled rate.

That is the backpressure story. I would say it plainly because “Kafka handles backpressure” is too hand-wavy for a staff interview. What I really mean is the broker decouples producer throughput from consumer throughput, and the consumer concurrency is intentionally bounded.

Step 6: Show the AWS cloud-native alternative

I also want this answer to show platform flexibility, because that usually comes up in a staff interview. If the company is deep in AWS, I would explain that the same pattern can be implemented with SNS and SQS instead of Kafka.

The mapping is pretty clean. SNS plays the broadcaster role. The Order Service publishes OrderCreated once to an SNS topic, and SNS fans that event out to separate SQS queues owned by Payment and Inventory. Each service then consumes from its own queue at its own pace. That preserves the two things I care about most here: decoupling and buffering.

I would also call out why SNS matters instead of sending directly to SQS. If Order publishes straight to one queue per consumer, the producer has to know every downstream service. SNS keeps Order decoupled. New consumers can subscribe later without changing Order at all.

Step 7: Show how eventual consistency appears in real state

I would not define eventual consistency in abstract terms. I would show it with a user-visible example: an order is accepted at time T1, payment finishes at T2, inventory reservation completes at T3, and the read model catches up at T4. In between those moments, different parts of the system can show slightly different but valid states.

That is not a bug. That is the cost of decoupling. The important architecture decision is to make those states explicit and expected.

PENDING -> PAYMENT_CONFIRMED -> INVENTORY_RESERVED -> CONFIRMED

or

PENDING -> PAYMENT_FAILED -> CANCELLED

I would also say that eventual consistency is acceptable here because ordering and payment are business workflows, not bank ledger entries. If the requirement were “all updates must commit atomically or not at all across services,” then I would need a very different design or a much tighter boundary.

Step 8: Cover poison messages and DLQs before the interviewer asks

Staff interviews usually turn into failure mode interviews pretty quickly. One of the first failure cases I would bring up myself is the poison message: maybe a bad schema, a missing field, or a code bug makes one event fail every time it is consumed.

If I do nothing, that message can be retried forever and block forward progress for that partition or consumer flow. So I would explain the retry policy and the DLQ together: retry a small number of times with backoff, then route the event to a dead-letter topic for inspection.

That last point is important. I have seen teams proudly mention DLQs and forget that somebody still has to monitor, triage, and replay them safely.

Step 9: Add the read side because staff interviews care about scale

The PRD also introduces a read-optimized side, which I think is a smart move for a staff-level answer. I would say that transactional writes and analytical reads have different needs. The Order Service should optimize for correctness and command latency. Reporting should optimize for query flexibility and throughput.

That is where projections help. Instead of joining across order, payment, and inventory tables on every read, I can project events into a denormalized view in Elasticsearch or another read-optimized store.

What I would say could be done better

This is the section that usually separates a decent answer from a stronger one. I would not pitch the PRD architecture as perfect. I would show how I would harden it.

Use the outbox pattern

The PRD shows the Order Service saving to the database and publishing to Kafka in sequence. In a real system, I would worry about the gap between those steps. If the DB write succeeds and the publish fails, the order exists but no event goes out. That is exactly where the transactional outbox pattern helps.

Design for idempotency explicitly

At-least-once delivery is common in event systems. That means Payment and Inventory should be able to process the same event more than once without double-charging or double-reserving. I would call out idempotency keys, dedupe tables, and business-safe consumers.

Be careful with choreography

The PRD leans toward event choreography, which is fine for a compact example. But once the business workflow gets more complex, I would at least discuss whether a saga orchestrator is needed for visibility and control. Choreography keeps services loosely coupled, but it can become hard to reason about when there are too many participants and compensating actions.

Own the event contract

I would want schema versioning, compatibility rules, and probably a schema registry. A lot of “Kafka problems” are really weak event governance problems.

Make observability first-class

I would add metrics for consumer lag, DLQ volume, retry rates, end-to-end order completion latency, and projection staleness. In interviews I like to say that asynchronous systems fail diagonally. You need telemetry that crosses services and topics or you will spend too much time guessing.

The concise answer I would leave the interviewer with

I would close by summarizing the logic in one chain: we return 202 Accepted because fast order intake matters more than synchronous completion; that pushes us toward asynchronous processing; once we do that, a broker like Kafka gives us decoupling, fan-out, buffering, and replay; because services now update independently, we accept eventual consistency and model state transitions explicitly; because failures are inevitable, we add bounded retries, DLQs, idempotent consumers, and read-side projections for scale.

That is the answer I would want to hear from a candidate too. Not just the tools, but the reasoning that connects one decision to the next.