Message Queues FAQ & Answers

Direct exchange routes messages to queues based on exact routing key match. Binding key must exactly equal message routing key. Use cases: task distribution with specific worker types, point-to-point messaging, routing by category or service name. Example: exchange 'tasks.direct', binding key 'order.processing', message with routing key 'order.processing' routes to order processing queue. Simple, predictable routing suitable for most basic message routing needs. Provides direct control over message destinations without complex patterns.

Fanout exchange broadcasts messages to all bound queues, completely ignoring routing keys. Every queue bound to the fanout exchange receives copy of every message. Use cases: event broadcasting, notifications, data synchronization across multiple services, real-time updates. Example: exchange 'events.fanout' with queues 'logger', 'analytics', 'cache' all bound - all receive every message. Fastest exchange type since no routing key matching required. Perfect for pub/sub patterns where all subscribers need all messages.

Topic exchange routes messages based on pattern matching with routing keys separated by dots. Wildcards: * matches exactly one word segment, # matches zero or more segments. Binding patterns like '.order.' or 'orders.#' enable flexible routing. Use cases: complex routing needs, hierarchical data, multi-criteria filtering. Example: routing key 'orders.payment.completed' matches bindings 'orders.*.completed' and 'orders.payment.#'. Most flexible exchange type supporting sophisticated message routing patterns. Essential for event-driven architectures with complex routing requirements.

Headers exchange routes messages based on header attributes instead of routing keys. Uses x-match header: 'all' requires all headers to match, 'any' requires at least one. Supports complex routing with multiple criteria. Use cases: content-based routing, A/B testing, feature flags, complex business rules. Example: message with headers {priority: 'high', region: 'us-east'} routes to queue bound with headers {priority: 'high'} and x-match: 'any'. Routing keys ignored. Perfect when routing decisions depend on message content rather than simple categorization.

Standard: at-least-once delivery (duplicates possible), best-effort ordering, unlimited throughput, 120K in-flight messages, cheaper. FIFO: exactly-once processing, strict message ordering, default 3K msg/sec with batching (300 without), high throughput mode supports 70K msg/sec, 20K in-flight messages, 5-minute content-based deduplication (SHA-256) or explicit deduplication ID, more expensive. Use Standard for high-volume, order-agnostic workloads. Use FIFO for transactional systems requiring strict ordering like payment processing, inventory management, financial transactions. Configure: aws sqs create-queue --queue-name my-fifo.fifo --attributes FifoQueue=true,ContentBasedDeduplication=true,FifoThroughputLimit=perMessageGroupId.

Automatic: message considered delivered immediately after sent (fire-and-forget), higher throughput but unsafe - if consumer crashes before processing, message lost. No built-in backpressure mechanism, risks consumer memory overflow. Manual: requires explicit ack (basic.ack), nack (basic.nack - supports bulk with multiple flag), or reject (basic.reject - single message only), provides delivery guarantees and flow control. Use manual ack for workloads requiring guaranteed processing. Configure: channel.basicConsume(queueName, autoAck=false, consumer). Manual mode enables prefetch limits preventing consumer overload: channel.basicQos(10) limits 10 unacked messages at a time.

SNS topic publishes message once, automatically replicated to multiple subscribed SQS queues for parallel asynchronous processing. Key benefits: full decoupling (producers and consumers operate independently), data persistence (SQS stores messages reliably until processed), automatic retries, dead letter queue support. Use cases: order processing (inventory, shipping, billing queues), media transcoding (multiple format queues), event broadcasting. Configure SNS filter policies for conditional routing based on message attributes (FilterPolicyScope: MessageAttributes) or payload (FilterPolicyScope: MessageBody). Subscribe: aws sns subscribe --topic-arn arn:aws:sns:region:account:topic --protocol sqs --notification-endpoint arn:aws:sqs:region:account:queue --attributes FilterPolicy='...'. Max 5 filter policy keys, 150 total value combinations per subscription.

AWS SQS: Dedicated FIFO queue type provides strict global message ordering, 3K-70K msg/sec depending on mode. Google Pub/Sub: No global ordering by default, best-effort prefers older messages. Supports opt-in per-key ordering using ordering keys - messages with same ordering key delivered in order received by service. Configure: enable message ordering on subscription (cannot change after creation), publish with ordering key (up to 1 KB, e.g., customer ID or database primary key). Ordering key throughput limit: 1 MBps per key. Regional guarantee only - subscribers must connect to same region as publisher. If message redelivered, all subsequent messages with same ordering key also redelivered. Enable: gcloud pubsub subscriptions create my-sub --topic=my-topic --enable-message-ordering.

Special exchange receiving undeliverable messages (dead letters). Messages dead-lettered when: 1) rejected/nacked with requeue=false (AMQP 0.9.1 basic.reject/nack or AMQP 1.0 rejected outcome), 2) message TTL expired, 3) queue length limit reached, 4) delivery limit exceeded. Configure: set x-dead-letter-exchange and optionally x-dead-letter-routing-key on queue. RabbitMQ 3.8+ tracks x-death parameter indicating how many times message traversed DLX - useful for retry counting. Example: channel.queueDeclare('main-queue', {durable: true, arguments: {'x-dead-letter-exchange': 'dlx.exchange', 'x-dead-letter-routing-key': 'failed.messages', 'x-message-ttl': 60000}}). Provides fault tolerance, troubleshooting insights, prevents poison messages from blocking normal queue operations. Essential for production reliability.

SQS delay queue delays message visibility for new messages before they can be consumed for the first time. Range: 0 to 15 minutes, default 0 seconds. Applied per-queue via DelaySeconds parameter. Use cases: future-dated tasks, scheduled processing, rate limiting, preventing immediate processing. Example: set DelaySeconds=300 for 5-minute delay before first consumer can process. Messages become visible after delay expires for all consumers. Does not affect redelivery visibility - only initial visibility. Different from message timers which apply per-message.

SQS visibility timeout hides messages after being received by consumer, preventing other consumers from processing same message. Range: 0 seconds to 12 hours, default 30 seconds. Consumer must delete message before timeout expires. If processing takes longer than timeout, consumer can extend visibility via ChangeMessageVisibility API. Use cases: preventing duplicate processing during long-running tasks, ensuring exactly-once semantics. Timeout reset each time message received, cannot exceed 12 hours total lifetime. Essential for reliable message processing patterns.

Yes (GA since December 2022). When enabled on subscriptions, Pub/Sub guarantees no redelivery after successful acknowledgment and no redelivery while message outstanding. Based on Pub/Sub-defined unique message ID, regional only (subscriber must connect to same region as service). Requirements: pull subscription type only (including StreamingPull API), client library versions: Python 2.13.6+, Java 1.139.0+, Go 1.25.1+, Node 3.2.0+, C# 3.2.0+, C++ 2.1.0+. Multiple valid deliveries (due to ack deadline expiration or nack), only latest acknowledgment ID can acknowledge - previous IDs fail. Compare AWS SQS: FIFO queues provide exactly-once, Standard queues at-least-once. Enable: gcloud pubsub subscriptions create my-sub --topic=my-topic --enable-exactly-once-delivery.

basic.reject: part of AMQP 0.9.1 specification, rejects single message only, no multiple flag support. basic.nack: RabbitMQ-specific protocol extension providing all basic.reject functionality plus bulk rejection. Set multiple=true flag to reject all unacknowledged delivered messages up to and including specified delivery_tag, complementing bulk acknowledgement semantics of basic.ack. Both support requeue parameter: true returns messages to queue head, false sends to DLX (if configured) or discards. Use basic.nack for bulk operations, basic.reject for AMQP broker compatibility. Example: channel.basicNack(deliveryTag, multiple=true, requeue=false) bulk rejects with DLX routing. Logically no need for reject since nack does everything reject can, but reject preferred for broader AMQP compatibility.

Default mode: 3,000 msg/sec with batching (300 batches of 10 messages), 300 msg/sec without batching. High throughput mode: 70,000 msg/sec without batching, even higher with batching. Standard queues: unlimited throughput. FIFO trades throughput for guaranteed ordering and exactly-once processing. Enable high throughput: aws sqs set-queue-attributes --queue-url <url> --attributes FifoThroughputLimit=perMessageGroupId,DeduplicationScope=messageGroup. High throughput mode requires per-message-group deduplication and ordering (not per-queue). Use message group IDs to parallelize processing while maintaining ordering within groups. Example: order processing with separate group IDs per customer maintains per-customer ordering while scaling across customers. Monitor with CloudWatch metrics: NumberOfMessagesSent, NumberOfMessagesReceived.

• (asterisk) matches exactly one word segment in specific position. # (hash) matches zero or more words (including multiple segments). Routing keys are dot-separated word lists. Examples: 'agreements...b.' matches only when first word is 'agreements' and fourth word is 'b'. 'agreements.eu.berlin.#' matches any routing keys beginning with 'agreements.eu.berlin'. 'logs..error' matches 'logs.api.error' but not 'logs.api.http.error'. 'logs.#.error' matches both. Implementation: RabbitMQ uses trie (tree data structure) storing each segment at next node, requires backtracking through trie to capture all wildcard matches. Use cases: hierarchical logging (app.*.error), regional routing (orders.us.#), multi-criteria event filtering. Binding: channel.queueBind(queueName, 'topic.exchange', 'orders.*.completed').

15 minutes (900 seconds) maximum delay, 0 seconds minimum (default). Delay applies when message first added to queue, affects initial visibility only. Configure per-queue: aws sqs create-queue --queue-name delayed-queue --attributes DelaySeconds=900. Per-message override: aws sqs send-message --queue-url <url> --message-body "..." --delay-seconds 300 (overrides queue default). Use cases: scheduled tasks, rate limiting, preventing immediate processing. For delays >15 minutes, workarounds: AWS Step Functions with Wait state (max 1 year), EventBridge Scheduler with Lambda trigger, SQS + Lambda循環 with visibility timeout extension, DynamoDB TTL with Streams trigger. Does not affect redelivery visibility - only first-time visibility. Different from visibility timeout (affects redelivery, max 12 hours).

12 hours (43,200 seconds) maximum from when message first received, 0 seconds minimum, 30 seconds default. Extending timeout doesn't reset the 12-hour limit - clock starts from first receive. After timeout expires, message becomes visible again for reprocessing. Configure per-queue: aws sqs create-queue --queue-name my-queue --attributes VisibilityTimeout=3600. Per-message extend during processing: aws sqs change-message-visibility --queue-url <url> --receipt-handle <handle> --visibility-timeout 600. Use ChangeMessageVisibility API for long-running tasks exceeding initial timeout. Example: initial 300s timeout, extend by 600s after 250s processing, total visibility 900s (still under 12h limit). Essential for preventing duplicate processing during long tasks. If processing exceeds 12 hours, redesign using Step Functions or chunked processing with intermediate checkpoints.

x-match special argument in binding arguments (not message headers) specifies matching logic. Values: 'all' requires all header pairs match (AND logic), 'any' requires at least one header pair match (OR logic), 'all-with-x' includes x-prefixed headers in all matching, 'any-with-x' includes x-prefixed headers in any matching. Important: headers beginning with 'x-' ignored by default 'all'/'any' matching - use 'all-with-x'/'any-with-x' to include them (common 2025 gotcha). Routes based on message header attributes, routing key ignored. Message matches if header value equals binding value. Example: bind queue with {priority: 'high', region: 'us-east', x-match: 'any'}, message with {priority: 'high'} routes successfully. Use cases: content-based routing, A/B testing, feature flags. Configure: channel.queueBind(queueName, 'headers.exchange', '', {priority: 'high', x-match: 'any'}).

5-minute (300-second) deduplication interval. SQS continues tracking MessageDeduplicationId even after message received and deleted. Duplicates introduced by producer removed within window. Two deduplication methods: 1) Content-based (automatic): enable ContentBasedDeduplication=true, SQS generates SHA-256 hash of message body (not attributes). 2) Explicit: provide MessageDeduplicationId token per message. If SQS accepted message with specific deduplication ID, subsequent messages with same ID acknowledged but not delivered within 5-minute window. After 5 minutes, messages with same deduplication ID can be processed again. Configure: aws sqs create-queue --queue-name my-queue.fifo --attributes FifoQueue=true,ContentBasedDeduplication=true. Per-message override: aws sqs send-message --queue-url <url> --message-body "..." --message-deduplication-id "unique-id-123". Essential for exactly-once processing in financial transactions, payment processing, inventory management.

Four conditions trigger dead-lettering: 1) Consumer rejects/nacks with requeue=false (AMQP 0.9.1 basic.reject/basic.nack or AMQP 1.0 rejected outcome), 2) Message TTL expired (x-message-ttl on queue or per-message expiration), 3) Queue length limit reached (x-max-length or x-max-length-bytes), 4) Delivery limit exceeded (quorum queues only, x-delivery-limit). Message republished to configured Dead Letter Exchange (x-dead-letter-exchange) with optional routing key override (x-dead-letter-routing-key). RabbitMQ 3.8+ adds x-death header parameter tracking death count, reason, queue, exchange, routing-keys - essential for retry counting and debugging. Original message properties preserved. Configure: channel.queueDeclare('main-queue', {arguments: {'x-dead-letter-exchange': 'dlx', 'x-message-ttl': 60000, 'x-max-length': 1000, 'x-delivery-limit': 3}}).

Filter policies (JSON objects) applied to SNS subscriptions conditionally route messages based on message attributes or payload. Two scopes via FilterPolicyScope attribute: 1) MessageAttributes (default) - filters on message attributes, 2) MessageBody - filters on payload content (introduced 2022). Constraints: max 5 filter policy keys, max 150 total value combinations (multiply values in each array), max 200 filter policies per topic, 10K per account. Changes take up to 15 minutes to fully take effect. Enables selective fanout where not every message sent to every downstream service, reduces costs and unnecessary processing. Example: subscribe SQS queue with filter {"event_type": ["order_created", "order_updated"], "region": ["us-east-1"]}. Configure: aws sns set-subscription-attributes --subscription-arn <arn> --attribute-name FilterPolicy --attribute-value '{"priority":["high"]}' --attribute-name FilterPolicyScope --attribute-value MessageAttributes.

SQS Standard queues support up to 120,000 in-flight messages per queue. In-flight messages are those received by consumers but not yet deleted (within visibility timeout). Configure with: aws sqs create-queue --queue-name my-queue --attributes VisibilityTimeout=30. High limit enables massive parallel processing for bursty workloads. Standard queues prioritize throughput over ordering, making them ideal for workloads that can handle occasional duplicates and don't require strict FIFO processing.

SQS FIFO queues support up to 20,000 in-flight messages per queue. In-flight messages are those received by consumers but not yet deleted (within visibility timeout). Configure with: aws sqs create-queue --queue-name my-fifo-queue --attributes FifoQueue=true,ContentBasedDeduplication=true. Lower limit balances ordering guarantees with throughput. FIFO queues ensure exactly-once processing and strict message order, perfect for transaction processing, inventory management, and financial systems where order and duplication prevention are critical.

Standard queues: 120,000 in-flight messages (6x higher). FIFO queues: 20,000 in-flight messages. Trade-off: Standard prioritizes throughput and scalability, FIFO prioritizes ordering guarantees and exactly-once processing. Monitor with: aws sqs get-queue-attributes --queue-url <url> --attribute-names ApproximateNumberOfMessagesNotVisible. Choose Standard for high-throughput, order-agnostic workloads. Choose FIFO for transactional systems requiring strict ordering and duplication prevention.

AMQP 1.0 feature (October 2024) enabling quorum queue consumers to add/update message annotations before requeueing or dead lettering. Key capabilities: 1) Custom dead letter routing - when dead lettering to headers exchange, consumer decides target queue via modified outcome, 2) Custom annotations - add metadata when requeueing for improved traceability and debugging, 3) Enhanced error handling - custom dead lettering event tracking instead of relying solely on x-death header. Supported in quorum queues only, not classic queues. Complements existing DLX patterns with consumer-driven routing decisions. Use cases: intelligent DLQ routing based on error type, retry counting with custom metadata, A/B testing failed message processing strategies. Example: consumer adds {error_type: 'validation', retry_count: 3} annotation before dead lettering, enabling routing to validation-specific DLQ. Official RabbitMQ blog (Oct 11, 2024) includes sample application demonstrating this feature.

Message not delivered and permanently lost. SNS is ephemeral publish-subscribe service without persistence or retry mechanism - messages exist only during delivery attempt. If no subscribers available, delivery fails, message discarded immediately. If subscriber endpoint unavailable or fails (HTTP 5xx, Lambda timeout, SQS throttle), SNS retries limited times then discards message. This is why SNS+SQS fanout pattern essential for production systems: SQS provides persistent storage, automatic retries (up to 14 days retention), dead letter queues for failed processing, visibility timeout for duplicate prevention. Pattern: SNS publishes once → multiple SQS queues subscribe → consumers process with guaranteed delivery. Alternative: configure SNS delivery status logging to CloudWatch for message delivery tracking and debugging. For critical messages requiring guaranteed delivery, always combine SNS with durable subscribers (SQS, Kinesis, HTTP endpoints with retries).

No global FIFO ordering by default - Pub/Sub uses best-effort ordering, often preferring older messages but no strict guarantee. Supports opt-in per-key ordering: messages with same ordering key delivered in order received by service within same region. Enable on subscription creation (cannot change after): gcloud pubsub subscriptions create my-sub --topic=my-topic --enable-message-ordering. Publisher must set ordering key (up to 1 KB, e.g., customer_id, user_id). Constraints: 1) Regional guarantee only - subscribers must connect same region as publisher, 2) Throughput limit 1 MBps per ordering key, 3) If message redelivered, all subsequent messages with same ordering key also redelivered even if already acknowledged. Compare AWS SQS: FIFO queues provide global ordering within queue, 3K-70K msg/sec depending on mode. Pub/Sub ordering more granular (per-key) vs SQS FIFO (per-queue or per-message-group).

Advanced pattern (2024) combining delayed retry queue (tier 1) with final dead letter queue (tier 2) for enhanced error handling. Architecture: Main Queue → Delay Queue (tier 1, x-message-ttl for retry delay) → Dead Letter Queue (tier 2, permanent failures). Tier 1: delayed processing accommodates transient failures (network issues, temporary service unavailability), messages wait with TTL then retry. Tier 2: prevents data loss for unrecoverable messages (validation errors, poison messages). RabbitMQ 3.8+ x-death parameter tracks retry count across DLX traversals, enabling retry limit logic. Important: messages only removed from queue head - cannot use single wait queue for variable back-off (10-min TTL message at head blocks 1-min TTL message behind). Solution: separate delay queues per retry interval or exponential backoff queues. Use cases: API retries with backoff, transient database connection failures, temporary downstream service outages. Example config: main queue with x-dead-letter-exchange='delay.exchange', delay queue with x-message-ttl=60000 and x-dead-letter-exchange='final.dlx'.

Message considered successfully delivered immediately after sent (fire-and-forget), before consumer actually processes it. Critical safety concerns: 1) If consumer's TCP connection or channel closes before successful processing, message lost permanently - no redelivery, 2) No built-in backpressure mechanism - server continues sending unlimited unprocessed messages, risking consumer memory overflow and crashes, 3) No delivery confirmation - cannot verify consumer actually processed message. Trades higher throughput (consumers can receive messages faster) for reduced delivery safety and reliability. Automatic acknowledgement mode unsuitable for all production workloads. Use cases limited to: non-critical logging, metrics, disposable data where occasional message loss acceptable. For guaranteed processing, always use manual acknowledgment (autoAck=false) combined with prefetch limits (channel.basicQos) for flow control. Manual mode ensures messages redelivered on consumer failure and prevents consumer overload.

Redrive policy configures automatic message transfer from source queue to dead letter queue (DLQ) after failed processing attempts. Key parameters: 1) deadLetterTargetArn - ARN of DLQ, 2) maxReceiveCount - how many times message can be received before moving to DLQ (range: 1-1,000, default 10). Each time consumer receives message, ReceiveCount increments by 1. When ReceiveCount > maxReceiveCount, SQS automatically moves message to DLQ for human analysis and debugging. Configure: aws sqs set-queue-attributes --queue-url <url> --attributes RedrivePolicy='{"deadLetterTargetArn":"arn:aws:sqs:region:account:dlq","maxReceiveCount":"3"}'. DLQ redrive to source queues (introduced 2021) enables moving messages back from DLQ to source after fixing consumer bugs. Prevents poison messages from blocking queue, enables inspection without data loss. Essential for production reliability - isolates problematic messages while maintaining normal queue operations.

Completely ignores routing keys - broadcasts copy of every message to every bound queue/stream/exchange regardless of routing key value. No routing logic, no pattern matching, pure broadcast. Fastest exchange type since no routing key processing overhead. Use cases: event broadcasting (all subscribers receive all events), real-time notifications (push to multiple clients), data synchronization (replicate across services), logging (send to multiple log processors). Example: exchange 'events.fanout' with bindings to queues 'logger', 'analytics', 'cache' - all three receive every message published. Configure: channel.exchangeDeclare('events.fanout', 'fanout', {durable: true}); channel.queueBind('logger-queue', 'events.fanout', ''). Binding key also ignored, use empty string. Perfect for pub/sub patterns where all subscribers need complete event stream. Contrast with Topic exchange (selective routing via wildcards) and Direct exchange (exact key matching).