Amazon SQS

Simple Queue Service — SQS

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.

• Amazon SQS is a highly available distributed queue system
• A queue is a temporary repository for messages awaiting for processing and acts as a buffer between the component producer and the consumer
• SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components.
• Amazon SQS fully managed and requires no administrative overhead and little configuration
• offers a reliable, highly-scalable, hosted queue for storing messages in transit between applications
• provides fault tolerant, loosely coupled, flexibility of distributed components of applications to send & receive without requiring each component to be concurrently available
• helps build distributed application with decoupled components
• supports the HTTP over SSL (HTTPS) and Transport Layer Security (TLS) protocols for security
• SQS provides two types of Queues — Standard & FIFO
• The message size is limited to 256KB. If you have a larger payload, you can store it on S3 and reference the object key in the message.

SQS FIFO Queue Features

SQS Standard Queue Features

• Redundant infrastructure
• offers reliable and scalable hosted queues for storing messages
• is engineered to always be available and deliver messages
• provides ability to store messages in a fail safe queue
• highly concurrent access to messages
• At-Least-Once delivery
• ensures delivery of each message at least once
• It guarantees the order of messages in the FIFO order.
• stores copies of the messages on multiple servers for redundancy and high availability
• might deliver duplicate copy of messages, if the servers storing a copy of a message is unavailable when you receive or delete the message and the copy of the message is not deleted on that unavailable server
• Applications should be designed to be idempotent with the ability to handle duplicate messages and not be adversely affected if it processes the same message more than once.

Message Attributes

• SQS message can contain up to 10 metadata attributes.
• take the form of name-type-value triples
• can be used to separate the body of a message from the metadata that describes it.
• helps process and store information with greater speed and efficiency because the applications don’t have to inspect an entire message before understanding how to process it

Message Sampling

• behavior of retrieving messages from the queue depends on whether short (standard) polling, the default behavior, or long polling is used.
• samples only a subset of the servers (based on a weighted random distribution) and returns messages from just those servers.
• A receive request might not return all the messages in the queue. But a subsequent receive request would return the message With Long polling,request persists for the time specified and returns as soon as the message is available thereby reducing costs and time the message has to dwell in the queue
• long polling doesn’t return a response until a message arrives in the message queue, or the long poll times out.
• makes it inexpensive to retrieve messages from the SQS queue as soon as the messages are available.
• might help reduce the cost of using SQS, as the number of empty receives are reduced

Batching

• SQS allows send, receive and delete batching, which helps club up to 10 messages in a single batch while charging price for a single message helps lower cost and also increases the throughput
• Configurable settings per queue
• All queues don’t have to be alike Order makes a best effort to preserve order in messages does not guarantee first in, first out delivery of messages
• can be handled by placing sequencing information within the message and performing the ordering on the client side

Loose coupling

• removes tight coupling between components
• provides the ability to move data between distributed components of the applications that perform different tasks without losing messages or requiring each component to be always available

Multiple writers and readers

• supports multiple readers and writers interacting with the same queue as the same time
• locks the message during processing, using Visibility Timeout, preventing it to be processed by any other consumer

Variable message size

• supports message in any format up to 256KB of text.
• messages larger than 256 KB can be managed using the S3 or DynamoDB, with SQS holding a pointer to the S3 object

Access Control

• Access can be controlled for who can produce and consume messages to each queue

Delay Queues

• delay queue allows the user to set a default delay on a queue such that delivery of all messages enqueued is postponed for that time duration

Dead Letter Queues

• Dead letter queue is a queue for messages that were not able to be processed after a maximum number of attempts
• useful to isolate messages that can’t be processed for later analysis.

PCI Compliance

• supports the processing, storage, and transmission of credit card data by a merchant or service provider, and has been validated as being PCI-DSS (Payment Card Industry — Data Security Standard) compliant

SQS Use Cases

• Work Queues Decouple components of a distributed application that may not all process the same amount of work simultaneously.

Buffer and Batch Operations

• Add scalability and reliability to the architecture and smooth out temporary volume spikes without losing messages or increasing latency

Request Offloading

• Move slow operations off of interactive request paths by enqueueing the request.

Fan-out

• Combine SQS with SNS to send identical copies of a message to multiple queues in parallel for simultaneous processing.

Parallel processing with “Fan Out” architecture

Figure 6. Fan out design pattern with S3, SNS, and SQS before sending to a Lambda function

To create a “fan out” style architecture where a single event is propagated to many destinations in parallel, SNS is combined with SQS. Configure your S3 event notification to use an SNS topic as its destination, as shown in Figure 6. You can then direct multiple subsequent processes to act on the same event. This is especially useful if you aim to do parallel processing on the same object in S3.

For example, if you wanted to process a source image into multiple target resolutions, you could create a Lambda function. The function will use the “fan-out” pattern to process all images at the same time, at each resolution. You could then subscribe an SQS queue to your SNS topics. This ensures that Event Notifications sent to SNS are verified as complete by SQS, once they’ve been processed by your Lambda function.

Auto Scaling

• SQS queues can be used to determine the load on an application, and combined with Auto Scaling, the EC2 instances can be scaled in or out, depending on the volume of traffic

How SQS Queues Works

• SQS allows queues to be created, deleted and messages can be sent and received from it
• SQS queue retains messages for four days, by default.
• Queues can be configured to retain messages for 1 minute to 14 days after the message has been sent.
• SQS can delete a queue without notification if any action hasn’t been performed on it for 30 consecutive days.
• SQS allows the deletion of the queue with messages in it

Queue and Message Identifiers

Queue URLs

• Queue is identified by a unique queue name within the same AWS account
• SQS assigns each queue with a Queue URL identifier for e.g. http://sqs.us-east-1.amazonaws.com/123456789012/queue2
• Queue URL is needed to perform any operation on the Queue

Message ID

• Message IDs are useful for identifying messages,
• Each message receives a system-assigned message ID that SQS returns to with the SendMessage response.
• To delete a message, the message’s receipt handle instead of the message ID is needed
• Message ID can be of is 100 characters max

Receipt Handle

• When a message is received from a queue, a receipt handle is returned with the message which is associated with the act of receiving the message rather then the message itself
• Receipt handle is required, not the message id, to delete a message or to change the message visibility
• If a message is received more than once, each time its received, a different receipt handle is assigned and the latest should be used always

Visibility timeout

• SQS does not delete the message once it is received by a consumer, because the system is distributed, there’s no guarantee that the consumer will actually receive the message (it’s possible the connection could break or the component could fail before receiving the message)
• Consumer should explicitly delete the message from the Queue once it is received and successfully processed
• As the message is still available on the Queue, other consumers would be able to receive and process and this needs to be prevented
• SQS handles the above behavior using Visibility timeout.
• SQS blocks the visibility of the message for the Visibility timeout period, which is the time during which SQS prevents other consuming components from receiving and processing that message
• Consumer should delete the message within the Visibility timeout. If the consumer fails to delete the message before the visibility timeout expires, the message is visible again for other consumers.
• Once Visible the message is available for other consumers to consume and can lead to duplicate messages.
• Visibility timeout considerations
• clock starts ticking once SQS returns the message
• should be large enough to take into account the processing time for each of the message
• default Visibility timeout for each Queue is 30 seconds and can be changed at the Queue level
• when receiving messages, a special visibility timeout for the returned messages can be set without changing the overall queue timeout using the receipt handle
• can be extended by the consumer, using ChangeMessageVisibility , if the consumer thinks it won’t be able to process the message within the current visibility timeout period. SQS restarts the timeout period using the new value
• a message’s Visibility timeout extension applies only to that particular receipt of the message and does not affect the timeout for the queue or later receipts of the messags.
• SQS has an 120,000 limit for the number of inflight messages per queue i.e. message received but not yet deleted and any further messages would receive an error after reaching the limit

Message Lifecycle

1. Component 1 sends Message A to a queue, and the message is redundantly distributed across the SQS servers.
2. When Component 2 is ready to process a message, it retrieves messages from the queue, and Message A is returned. While Message A is being processed, it remains in the queue but is not returned to subsequent receive requests for the duration of the visibility timeout.
3. Component 2 deletes Message A from the queue to avoid the message being received and processed again once the visibility timeout expires.

SQS Buffered Asynchronous Client

• Amazon SQS Buffered Async Client for Java provides an implementation of the AmazonSQSAsyncClient interface and adds several important features:
• Automatic batching of multiple SendMessage, DeleteMessage, or ChangeMessageVisibility requests without any required changes to the application
• Prefetching of messages into a local buffer that allows the application to immediately process messages from SQS without waiting for the messages to be retrieved
• Working together, automatic batching and prefetching increase the throughput and reduce the latency of the application while reducing the costs by making fewer SQS requests.

SQS Security and reliability

• SQS stores all message queues and messages within a single, highly-available AWS region with multiple redundant Availability Zones (AZs)
• SQS supports the HTTP over SSL (HTTPS) and Transport Layer Security (TLS) protocols.
• SQS supports Encryption at Rest. SSE encrypts messages as soon as SQS receives them and decrypts messages only when they are sent to an authorized consumer.
• SQS also supports resource-based permissions

SQS Design Patterns

Priority Queue Pattern

1. Use SQS to prepare multiple queues for the individual priority levels.
2. Place those processes to be executed immediately (job requests) in the high priority queue.
3. Prepare numbers of batch servers, for processing the job requests of the queues, depending on the priority levels.
4. Queues have a message “Delayed Send” function, which can be used to delay the time for starting a process.

SQS Job Observer Pattern

1. Enqueue job requests as SQS messages.
2. Have the batch server dequeue and process messages from SQS.
3. Set up Auto Scaling to automatically increase or decrease the number of batch servers, using the number of SQS messages, with CloudWatch, as the trigger to do so.

Amazon SQS short and long polling

Amazon SQS provides short polling and long polling to receive messages from a queue. By default, queues use short polling.

With short polling, the ReceiveMessage request queries only a subset of the servers (based on a weighted random distribution) to find messages that are available to include in the response. Amazon SQS sends the response right away, even if the query found no messages.

With long polling, the ReceiveMessage request queries all of the servers for messages. Amazon SQS sends a response after it collects at least one available message, up to the maximum number of messages specified in the request. Amazon SQS sends an empty response only if the polling wait time expires.

The following sections explain the details of short polling and long polling.

Topics

• Consuming messages using short polling
• Consuming messages using long polling
• Differences between long and short polling

Consuming messages using short polling

When you consume messages from a queue using short polling, Amazon SQS samples a subset of its servers (based on a weighted random distribution) and returns messages from only those servers. Thus, a particular ReceiveMessage request might not return all of your messages. However, if you have fewer than 1,000 messages in your queue, a subsequent request will return your messages. If you keep consuming from your queues, Amazon SQS samples all of its servers, and you receive all of your messages.

The following diagram shows the short-polling behavior of messages returned from a standard queue after one of your system components makes a receive request. Amazon SQS samples several of its servers (in gray) and returns messages A, C, D, and B from these servers. Message E isn’t returned for this request, but is returned for a subsequent request.

Consuming messages using long polling

When the wait time for the ReceiveMessage API action is greater than 0, long polling is in effect. The maximum long polling wait time is 20 seconds. Long polling helps reduce the cost of using Amazon SQS by eliminating the number of empty responses (when there are no messages available for a ReceiveMessage request) and false empty responses (when messages are available but aren't included in a response). For information about enabling long polling for a new or existing queue using the Amazon SQS console, see the Configuring queue parameters (console). For best practices, see Setting up long polling.

Long polling offers the following benefits:

• Reduce empty responses by allowing Amazon SQS to wait until a message is available in a queue before sending a response. Unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage action. In rare cases, you might receive empty responses even when a queue still contains messages, especially if you specify a low value for the ReceiveMessageWaitTimeSeconds parameter.
• Reduce false empty responses by querying all — rather than a subset of — Amazon SQS servers.
• Return messages as soon as they become available.

For information about how to confirm that a queue is empty, see Confirming that a queue is empty.

How reliable is the storage of my data in Amazon SQS?

Amazon SQS stores all message queues and messages within a single, highly-available AWS region with multiple redundant Availability Zones (AZs), so that no single computer, network, or AZ failure can make messages inaccessible. For more information, see Regions and Availability Zones in the Amazon Relational Database Service User Guide.

Reference :

https://aws.amazon.com/blogs/architecture/modernized-database-queuing-using-amazon-sqs-and-aws-services/

SQS Design Patterns Archives - Jayendra's Cloud Certification Blog

https://medium.com/awesome-cloud/aws-difference-between-sqs-and-sns-61a397bf76c5