Async Processing & Messaging

Message queue (RabbitMQ, SQS): task-based — one consumer processes each message, and the message is deleted after consumption. Use for: background jobs, email sending, image processing, webhook delivery. Event streaming (Kafka): an event log that multiple consumer groups read independently, with messages retained. Use for: audit logs, analytics pipelines, event sourcing, and feeding multiple downstream systems (payments → analytics + fraud detection + notifications simultaneously). The key question: do you need replay, multiple independent consumers, or long retention? Kafka. Just want reliable task execution? A queue.

Design consumers to be idempotent. Three techniques: (1) Unique constraint: try the INSERT, catch the duplicate-key error, treat it as success. (2) Check-then-process: SELECT first, only process if not seen before — but beware race conditions; use SELECT FOR UPDATE or a Redis SETNX lock. (3) Idempotency key: the client provides a unique ID with each request; the server stores processed IDs in Redis or the DB and deduplicates. The pattern: if redis.setnx(f"processed:{msg_id}", 1, ex=3600): process(msg) — only the first delivery is processed.

(1) Priority queues: separate queues for CRITICAL (account security), HIGH (order updates), LOW (marketing). CRITICAL uses SQS FIFO or Kafka priority partitions and is processed first. (2) Fanout: one event ("order_placed") → fanout service → separate queues per channel (email, push, SMS), each with dedicated consumers calling SES, FCM, and Twilio. (3) Rate limiting: email providers have per-second limits — use a token bucket limiter per provider. (4) Retry: SQS visibility timeout plus a dead letter queue for failures. (5) Scale check: 100M/day ≈ 1,160/sec average, ~5,000/sec peak — Kafka handles this trivially with 10 partitions × 10 consumers per channel.

Event-Driven Architecture (EDA) is a software architecture pattern where system components communicate through events rather than direct synchronous calls. Instead of services invoking each other directly, they publish events when something happens, and other services listen for and react to those events asynchronously.

Differences from Request-Response: Decoupling — in REST-based systems, services depend on each other directly; in EDA services are decoupled and interact only through events. Scalability — EDA scales better as new consumers can subscribe to events without modifying producers. Asynchronous Processing — traditional architectures require waiting for responses; EDA enables non-blocking workflows.

Pub-Sub (RabbitMQ, AWS SNS, Redis Pub/Sub): One-to-many event distribution. Events are transient — once consumed, they are gone. No strict ordering guarantee. Best for real-time notifications where consumers need only the latest event.

Event Streaming (Apache Kafka, AWS Kinesis): Events are stored and can be replayed. Maintains strict event order within partitions. Events persist for later processing. Best for processing historical data, event sourcing, and audit logs where consumers may need to replay past events.

1. Event Producers: Emit events when something happens (e.g., a user places an order, a sensor reports a reading).

2. Event Brokers: Middleware that routes events — examples: Kafka, RabbitMQ, AWS EventBridge. Acts as the communication backbone between producers and consumers.

3. Event Consumers: Services that process events asynchronously and react (update a database, send a notification, trigger a workflow).

4. Event Store (optional): A persistent log of all events used for auditing, debugging, or event replay to reconstruct past system state.

Challenges: Eventual Consistency — events propagate asynchronously so data may not be updated instantly across all services; Event Ordering — ensuring correct sequence in distributed systems; Debugging Complexity — events flow across many services making it hard to trace issues; Handling Failures — consumers may fail while processing events.

Handling Eventual Consistency: Use idempotent operations to avoid duplicate updates; implement Sagas (orchestration/choreography patterns) to ensure business logic correctness; leverage Event Sourcing to reconstruct system state from the ordered event log if inconsistency is detected.

Partitioning: Kafka uses partitions to ensure ordering within each partition — events for the same key (e.g., same user ID) always go to the same partition.

Event Versioning: Maintain a version number in events and process them sequentially, rejecting out-of-order versions.

Global Ordering Service: Use a dedicated ordering service that assigns sequence numbers to events before they reach consumers.

Deduplication: Implement unique event IDs so consumers can detect and discard already-processed or out-of-order duplicate events.

A Dead-Letter Queue (DLQ) is a separate queue where failed or unprocessable messages are sent after exhausting retry attempts.

Importance: Prevents infinite retries — avoids a faulty message being retried indefinitely and blocking the queue; Facilitates debugging — developers can inspect dead-lettered events to identify root causes; Ensures reliability — prevents faulty messages from blocking the main queue and causing backpressure across the system.

In Kafka, a dedicated topic stores dead-lettered messages for later analysis and reprocessing after the root cause is fixed.

Apache Kafka: Event Streaming type. Stores events for replay with guaranteed ordering within partitions. Highly scalable with partitions. Best for large-scale event processing and analytics pipelines.

RabbitMQ: Pub-Sub type. Transient messages (once consumed, gone). No strict ordering guarantee. Scales horizontally. Best for real-time messaging and task queues where replay is not needed.

AWS EventBridge: Managed Event Bus. No persistence. No strict ordering. Scales with AWS infrastructure. Best for cloud-native event integration between AWS services and external SaaS applications.

Idempotency ensures that processing the same event multiple times does not produce unintended side effects — critical in at-least-once delivery systems.

Use Unique Event IDs: Track processed event IDs in a database or cache to prevent re-processing the same event twice. Store Processed States: Maintain a "processed" flag in the database for each event. Ensure Business Logic is Idempotent: Avoid unconditional increments or timestamps — check current state first. Leverage Broker Deduplication: Kafka allows deduplication via log compaction; SQS has a deduplication ID feature.

E-commerce Order Processing System: When a customer places an order, multiple actions must happen: the Order Service records the order, the Payment Service processes payment, the Inventory Service updates stock, and the Notification Service sends a confirmation.

Using EDA, each service listens for order events asynchronously, with no direct dependencies on each other. Benefits: scalability (each service scales independently), fault isolation (a payment failure does not crash inventory), and flexibility (new services like fraud detection or analytics can be added by simply subscribing to order events without modifying existing code).

When events change over time (adding/removing fields), breaking changes can occur. Best practices:

Backward Compatibility: Ensure new consumers can still process older event formats. Versioning: Include a schema_version field in events and maintain handlers for different versions. Schema Registry: Use tools like Kafka Schema Registry (with Avro or Protobuf) to validate schema changes before they reach consumers. Field Deprecation: Instead of removing fields immediately, mark them as deprecated and stop producing them gradually over multiple releases. Event Transformation Layer: Introduce an intermediary service that translates older event schemas to the new format for legacy consumers.

Asynchronous messaging decouples the sender and receiver, allowing them to operate independently. This improves scalability because producers don't wait for consumers; resilience because messages are queued even if the consumer is slow or down; performance because users get a fast response while background tasks (e.g., sending emails, processing orders) are offloaded; and loose coupling because systems can evolve independently without tight runtime dependencies. A classic example: sending a confirmation email after a purchase — there is no need to delay the user response for email delivery.

RabbitMQ uses a queue-based push model with FIFO ordering per queue. Messages are deleted once consumed, making it ideal for real-time, transactional workloads that need per-message acknowledgment and complex routing (e.g., order processing, real-time updates). Kafka uses a log-based pull model with per-partition ordering and configurable retention — messages are kept even after consumption, enabling replay. Kafka is best for high-throughput data pipelines, event sourcing, and analytics (e.g., tracking user behavior at scale). RabbitMQ delivers at-most-once or at-least-once guarantees; Kafka can achieve exactly-once with the right configuration.

At-most-once: The message is sent once and not retried on failure. It is fast and simple but can lose data, making it appropriate for non-critical logging or telemetry. At-least-once: The message is retried until acknowledged, which is the default in most systems. It may result in duplicates so consumers must be idempotent. Suitable for order placement and email dispatching. Exactly-once: The message is delivered exactly once with no duplication or loss. It is complex and expensive to implement but required for critical financial transactions such as bank transfers and billing systems.

Queues act as a buffer between producers and consumers, so producers can send messages quickly without waiting for slow consumers. Consumers can be scaled horizontally and independently of producers to keep up with load. For fault tolerance: if a consumer crashes, messages remain safely in the queue until it recovers or another consumer instance takes over. Queues also smooth traffic spikes — during high-traffic events like Black Friday, the queue absorbs the surge and lets backend systems catch up at their own pace without being overwhelmed.

Common problems include queue backlog when consumers can't keep up, increased message delay leading to latency spikes, message duplication from retries, and resource exhaustion from CPU and memory strain caused by deep backlogs. Mitigations: scale consumers horizontally using auto-scaling groups; optimize message processing to reduce per-message work; use dead-letter queues (DLQ) to isolate and inspect poison messages that repeatedly fail; and implement rate limiting and backpressure mechanisms to signal producers to slow down when the system is under strain.

When a user places an order, the frontend service acts as a producer and publishes an order message to a queue. An Order Service (consumer) reads from the queue, validates stock, reserves inventory, and triggers downstream services for payment and confirmation emails — each of which may publish their own events in an event-driven chain. The queue provides value by decoupling payment and inventory logic from the frontend (users get an instant response), handling retries automatically on transient failures, and allowing each downstream service to scale independently based on its own load.

Assign each message a unique message ID (e.g., Order ID or broker-generated Message ID) and maintain a processed-message store (in Redis or a database table) that tracks messages already handled. Before processing a message, check whether its ID already exists in the store — if so, skip it. Make business operations inherently idempotent: for example, "mark order as paid" should be a no-op if the order is already marked paid. Leverage built-in deduplication features where available (e.g., AWS SQS FIFO queues have a deduplication window). This is essential in at-least-once delivery scenarios where retries are expected.