suqs

Sasquatch Message Queue

A lightweight, SQS-compatible message queue service backed by SQLite. Single binary, zero external dependencies beyond the database file.

Quick Start

# Run with file-backed storage
./sasquatch --port 8080

# Run with in-memory storage (data lost on restart)
./sasquatch --memory --port 8080

Command Line Options

Flag	Default	Description
--port	8080	Port to listen on
--host	localhost	Host to bind to
--memory	false	Use in-memory SQLite (no persistence)
--max-queue-length	5000	Maximum messages per queue
--max-message-size	256	Maximum message size in KB (max: 10240)
--version		Print version and exit
--help		Print help and exit
--config		Path to .ini configuration file
--generate-config		Print a sample .ini config to stdout and exit

CLI flags override config file values. Config file values override built-in defaults.

---

Configuration File

All tuneable system parameters can be set via an .ini file. Generate a sample with all defaults:

./sasquatch --generate-config > sasquatch.ini

Then start with it:

./sasquatch --config sasquatch.ini

Config Reference

[server]
port = 8080                          # 1-65535
host = localhost
memory = false                       # true/false/yes/no/1/0

[queue]
max_queue_length = 5000              # >= 1
max_message_size_kb = 256            # 1-10240
max_request_body_size_kb = 1024      # >= 1
default_visibility_timeout = 30      # 0-43200 seconds
default_max_receives = 4             # >= 1
poison_max_receives = 100            # >= default_max_receives
default_retention_period = 345600    # 60-1209600 seconds (4 days default)
default_wait_time_seconds = 20       # 0-20
default_dedup_time_period = 300      # 1-43200 seconds (5 min default)
default_priority = 100               # >= 0

[cleanup]
poison_cleanup_interval = 60         # seconds, >= 1
retention_cleanup_interval = 600     # seconds, >= 1
dedup_cleanup_interval = 60          # seconds, >= 1

[database]
reader_pool_size = 4                 # 1-64

The config file is validated on startup. If any value is out of range, an unknown key is present, or a duplicate key is found, the service prints all errors and exits with a non-zero status. Section headers are optional — all keys are matched by name regardless of section.

---

API Reference

All endpoints enforce HTTP methods (POST or GET as documented). Requests with the wrong method receive a 405 Method Not Allowed response.

All POST endpoints accept and return JSON. All errors are returned as:

{
  "error": {
    "code": "ErrorCode",
    "message": "Human-readable description"
  }
}

Internal errors return a generic message to clients. Details are logged server-side only.

POST /send

Send a message to a queue. Queues are created implicitly on first message.

Request:

{
  "queue_name": "my-queue",
  "message_body": "Hello, world!",
  "priority": 100,
  "delay_seconds": 0,
  "message_retention_period": 345600,
  "message_deduplication_id": "abc123",
  "message_deduplication_time_period": 300,
  "message_attributes": {
    "content_type": "application/json",
    "correlation_id": "req-12345"
  }
}

Field	Required	Default	Description
queue_name	Yes		Alphanumeric, hyphens, underscores
message_body	Yes		Message content string (up to max-message-size)
priority	No	100	Higher = dequeued first (0–1000). Explicit 0 is respected
delay_seconds	No	0	Seconds before message becomes visible (0–43200)
message_retention_period	No	345600 (4 days)	Seconds before auto-deletion (60–1209600)
message_deduplication_id	No		Alphanumeric, max 128 chars. Rejects duplicates within the time period
message_deduplication_time_period	No	300 (5 min)	Seconds the dedup ID is enforced (max 43200)
message_attributes	No		Key-value metadata (max 10 keys, key max 64 chars, value max 1024 chars)

Response (200):

{
  "message_id": "550e8400-e29b-41d4-a716-446655440000",
  "md5_of_message_body": "65a8e27d8879283831b664bd8b7f0ad4",
  "sequence_number": 42
}

Error codes: ValidationError (400), QueueFull (503), MessageSizeExceeded (413), DuplicateMessage (409)

---

POST /send_batch

Send up to 10 messages in a single request. Each message is validated independently — one bad message does not prevent the others from being sent. All successful messages are inserted in a single transaction (one fsync instead of 10).

Request:

{
  "messages": [
    {"queue_name": "orders", "message_body": "order-1", "priority": 100},
    {"queue_name": "orders", "message_body": "order-2", "priority": 50, "message_attributes": {"source": "web"}},
    {"queue_name": "alerts", "message_body": "disk-full", "priority": 200}
  ]
}

Field	Required	Description
messages	Yes	Array of 1–10 SendRequest objects (same fields as /send)

Response (200):

{
  "successful": [
    {"index": 0, "message_id": "...", "md5_of_message_body": "...", "sequence_number": 1},
    {"index": 2, "message_id": "...", "md5_of_message_body": "...", "sequence_number": 1}
  ],
  "failed": [
    {"index": 1, "error": {"code": "DuplicateMessage", "message": "..."}}
  ]
}

The index field corresponds to the position in the input messages array. Messages can be sent to different queues in the same batch. Queue length is checked once per queue and cached across the batch.

---

POST /receive

Receive one or more messages with long polling. Messages become invisible to other consumers for the duration of the visibility timeout. If not deleted within that window, they become visible again.

Request:

{
  "queue_name": "my-queue",
  "visibility_timeout": 30,
  "wait_time_seconds": 20,
  "max_number_of_messages": 1,
  "max_receives": 4,
  "dead_letter_queue_name": "my-queue-dlq",
  "peek": false,
  "shuffle": false
}

Field	Required	Default	Description
queue_name	Yes		Queue to receive from
visibility_timeout	No	30	Seconds message is hidden after receive (0–43200). Explicit 0 means message stays visible after receive
wait_time_seconds	No	20	Long-poll duration if no messages available (0–20). Explicit 0 disables long-polling (returns immediately)
max_number_of_messages	No	1	Number of messages to receive (1–10)
max_receives	No	4	Max times a message can be received before poison handling (1–10)
dead_letter_queue_name	No		Queue to move poison messages to (must differ from queue_name)
peek	No	false	Read-only mode: returns the next message without changing visibility, receive count, or issuing a receipt handle
shuffle	No	false	Return a random visible message instead of priority/FIFO order

Response (200):

{
  "messages": [
    {
      "message_id": "550e8400-e29b-41d4-a716-446655440000",
      "receipt_handle": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
      "message_body": "Hello, world!",
      "md5_of_message_body": "65a8e27d8879283831b664bd8b7f0ad4",
      "priority": 100,
      "approximate_receive_count": 1,
      "approximate_first_receive_timestamp": 1700000000,
      "sent_timestamp": 1700000000,
      "sequence_number": 42,
      "message_attributes": {
        "content_type": "application/json",
        "correlation_id": "req-12345"
      }
    }
  ]
}

If no messages are available after the wait time, returns {"messages": []}.

Peek Mode

When peek is true, the message is returned read-only — no visibility change, no receive count increment, no receipt handle. The message remains visible to other consumers. Peek always returns at most one message (repeated peeks return the same message). Cannot be combined with dead_letter_queue_name.

Shuffle Mode

When shuffle is true, a random visible message is selected instead of following priority/FIFO ordering. Can be combined with peek for a random read-only sample. Works normally with visibility semantics when peek is false.

---

POST /delete

Delete one or more messages by receipt handle. Call this after successfully processing a message.

Request:

{
  "receipt_handles": ["7c9e6679-7425-40de-944b-e07fc1f90ae7"]
}

Field	Required	Description
receipt_handles	Yes	Array of UUIDs (1–10)

Response (200):

{
  "deleted": 1
}

Error codes: NotFound (404) if no messages matched

Deduplication note: Messages sent with a message_deduplication_id are soft-deleted — the body is cleared but the row remains until the deduplication time period expires, preventing the same dedup ID from being reused.

---

POST /change_visibility

Change the visibility timeout of messages by receipt handle. Affects messages regardless of whether they are currently in-flight or have become visible again, as long as the receipt handle matches (a re-received message gets a new receipt handle, so stale handles naturally stop matching).

Request:

{
  "receipt_handles": ["7c9e6679-7425-40de-944b-e07fc1f90ae7"],
  "visibility_timeout": 60
}

Field	Required	Description
receipt_handles	Yes	Array of UUIDs (1–10)
visibility_timeout	Yes	New timeout in seconds from now (-43200 to 43200)

A negative value makes the message visible immediately (releases it back to the queue).

Response (200):

{
  "result": "ok"
}

---

POST /delete_all

Purge all messages from a queue.

Request:

{
  "queue_name": "my-queue"
}

Use "queue_name": "*" to delete all messages across all queues.

---

POST /queue_length

Get the number of visible and in-flight messages in a queue.

Request:

{
  "queue_name": "my-queue"
}

Response (200):

{
  "queue_name": "my-queue",
  "count": 42,
  "in_flight": 3
}

count is the number of visible (available) messages. in_flight is the number of messages currently being processed (received but not yet deleted or timed out).

---

GET /queues

List all queues with visible and in-flight message counts. No request body.

Response (200):

[
  {"queue_name": "orders", "count": 150, "in_flight": 5},
  {"queue_name": "notifications", "count": 12, "in_flight": 0}
]

---

GET /health

Response (200):

{
  "status": "ok",
  "version": "7"
}

---

GET /stats

Returns an HTML page with request counters. Stats reset on service restart.

---

Legacy Endpoints

These exist for backward compatibility:

• POST /enqueue — Send via query params (queue_name, priority) with raw body as the message. Returns JSON.
• POST /dequeue — Alias for /receive.

---

Core Concepts

Message Lifecycle

Send → [delay period] → Visible → Receive → Invisible → Delete
                                      ↑          |
                                      |          (timeout expires)
                                      +----------+

1. Send: Message enters the queue. If delay_seconds is set, it's invisible until the delay expires.
2. Receive: Message becomes invisible for visibility_timeout seconds. A receipt_handle is returned.
3. Process: Your application processes the message.
4. Delete: Call /delete with the receipt handle. If you don't delete within the visibility timeout, the message becomes visible again and can be received by another consumer.

Priority

Messages are ordered by priority DESC, sent_at ASC within a queue. Higher priority numbers are dequeued first. Default priority is 100. Priority must be in the range 0–1000. An explicit priority of 0 is respected (not treated as "use default"). Within the same priority level, messages are delivered in FIFO order.

LIFO Queues

Queue names ending in _lifo (e.g., jobs_lifo) automatically reverse the ordering within each priority level, delivering the most recently sent message first.

Peek

Setting "peek": true on a receive request returns the next message without any side effects — no visibility change, no receive count increment, no receipt handle. Useful for inspecting the head of a queue, monitoring, or debugging without affecting consumers. Peek uses the read-only connection pool and does not block writers.

Shuffle / Random Receive

Setting "shuffle": true on a receive request selects a random visible message instead of following priority/FIFO order. This is useful for workloads where you want to distribute processing randomly rather than in strict order. Can be combined with peek for a random read-only sample.

Message Attributes

Messages can carry up to 10 key-value string attributes as structured metadata alongside the body. Attributes are useful for routing, filtering, content type signaling, correlation IDs, or any metadata that consumers need without parsing the message body.

Limits: max 10 attributes per message, keys 1–64 characters, values up to 1024 characters. Attributes are stored as JSON in the database and returned as-is on receive. If no attributes are set, the field is omitted from the response.

Dead Letter Queue

When a message has been received more than max_receives times without being deleted, it's considered a poison message. If dead_letter_queue_name is specified on the receive request, the message is moved to that queue with its receive count reset to zero. If no DLQ is specified, poison messages are deleted.

The DLQ is just a regular queue — you can receive from it, inspect failed messages, and reprocess them. The DLQ name must differ from the source queue name. If the DLQ is full (at max_queue_length), poison messages are deleted instead of moved, and a warning is logged.

Poison messages are cleaned up in bulk before each receive operation, eliminating the need for per-message retry loops.

Message Deduplication

When sending with a message_deduplication_id, the system rejects any subsequent send with the same ID (scoped to the same queue) until the deduplication time period expires. This prevents duplicate processing when a producer retries after a timeout.

The dedup ID is enforced even after the message is deleted — the system retains a tombstone record (with cleared body) until the dedup period expires. After expiry, the tombstone is cleaned up and the same dedup ID can be reused.

Visibility Timeout and ChangeVisibility

After receiving a message, you have visibility_timeout seconds to process and delete it. If processing is taking longer than expected, call /change_visibility to extend the window. If you know you can't process the message, call /change_visibility with a negative timeout to release it back immediately.

/change_visibility matches on receipt handle only. If a message's visibility has expired and it has been re-received by another consumer (which assigns a new receipt handle), the old handle naturally stops matching.

Message Retention

Messages are automatically deleted after their retention period expires (default 4 days, configurable per-message up to 14 days). A background task runs every 10 minutes to clean expired messages.

Sequence Numbers

Each queue maintains an independent, monotonically increasing sequence number counter. The sequence number is assigned at send time and is unique within a queue. Unlike the internal database row ID, sequence numbers are stable — they don't change when messages are moved to a DLQ. Sequence counters are never deleted, ensuring the monotonic guarantee holds even if a queue fully drains and later receives new messages.

---

Error Codes

Code	HTTP Status	Description
InvalidRequest	400	Malformed JSON or missing required fields
ValidationError	400	Field validation failed
MethodNotAllowed	405	Wrong HTTP method for the endpoint
NotFound	404	No messages matched the receipt handles
DuplicateMessage	409	Message deduplication ID already exists
MessageSizeExceeded	413	Message body exceeds configured max
QueueFull	503	Queue has reached max-queue-length
InternalError	500	Database or server error (details logged server-side only)

All errors use a typed error system internally (sentinel errors with errors.Is/errors.As). Internal error details are never exposed to clients.

---

Architecture

SQLite Configuration

The database is opened with production-tuned pragmas:

Pragma	Value	Purpose
journal_mode	WAL	Concurrent reads alongside writes
busy_timeout	5000ms	Wait instead of failing on lock contention
synchronous	NORMAL	Safe with WAL, avoids fsync on every commit
cache_size	64MB	Larger page cache reduces disk reads
foreign_keys	ON	Referential integrity
txlock	IMMEDIATE	Prevents read-to-write lock upgrade deadlocks

Reader/Writer Split

The system uses two separate sql.DB connection pools:

• Writer (MaxOpenConns=1): All INSERT, UPDATE, DELETE operations go through a single connection. This is mandatory for SQLite — it only supports one writer at a time. Serializing writes at the connection level avoids SQLITE_BUSY errors and eliminates contention.

• Reader (MaxOpenConns=4): Read-only queries (/queue_length, /queues, peek) use a separate pool with multiple connections. WAL mode allows these reads to execute concurrently with writes and with each other, utilizing multiple CPU cores.

This is the recommended pattern for SQLite in Go. It maximizes throughput for mixed read/write workloads without the complexity of database-per-queue sharding.

Request Body Limits

All JSON endpoints are limited to 1MB request bodies via http.MaxBytesReader (with the ResponseWriter passed so the Connection: close header is set on overflow). The /send endpoint allows up to max-message-size + 4KB for JSON overhead. The /send_batch endpoint allows up to max-message-size * 10 + 4KB.

HTTP Method Enforcement

All endpoints enforce their expected HTTP method (POST or GET). Requests with incorrect methods receive a 405 Method Not Allowed JSON error response.

Per-Queue Notifications

Long-poll consumers subscribe to notifications for their specific queue. When a message is sent to a queue, only consumers waiting on that queue are woken — sends to queue A do not trigger spurious wakeups on queue B consumers. A 1-second ticker provides a safety-net fallback.

Client Disconnect Detection

Long-poll receive requests select on r.Context().Done() alongside the timeout and notification channels. When a client disconnects mid-poll, the server stops querying immediately instead of burning CPU for the remaining wait time.

Graceful Shutdown

The server handles SIGINT and SIGTERM. On shutdown: the cleanup goroutine's context is cancelled (stopping background tasks cleanly), in-flight HTTP requests get up to 10 seconds to complete, then the database connections are closed.

Request Logging

All requests are logged via structured logging middleware with method, path, remote address, and duration in milliseconds.

Background Tasks

Three background tasks run on timers, all respecting a cancellation context for clean shutdown:

• Retention cleanup (every 10 min): Deletes messages past their retention timestamp
• Dedup cleanup (every 1 min): Deletes tombstone records whose dedup period has expired; clears expired dedup IDs on active messages so they can be reused
• Poison safety net (every 1 min): Deletes messages with receive_count >= poison_max_receives as a backstop

Sequence counters are intentionally never cleaned up — they are one small row per queue name ever used, and cleaning them would break the monotonic sequence guarantee if a queue drains and refills.

Indexes

Index	Purpose
idx_dequeue	Composite covering index for receive queries
idx_receipt	Partial index on receipt_handle (WHERE NOT NULL) for delete/change_visibility
idx_expires	For retention cleanup task
idx_poison	For poison message safety net
idx_dedup	Unique partial index enforcing deduplication
idx_dedup_exp	For dedup cleanup task

Structured Logging

Uses log/slog (Go 1.21+ stdlib) for structured JSON-compatible log output. All log entries include key-value pairs for machine parsing.

Security

• Internal error messages (SQLite errors, file paths, stack traces) are never returned to clients. The client receives a generic "an internal error occurred" message; full details are logged server-side.
• HTTP methods are enforced on all endpoints.
• Batch operations (delete, change_visibility) use parameterized SQL IN clauses rather than per-item queries.
• Request body size limits are enforced on all endpoints, including legacy /enqueue.
• All user-provided numeric parameters are range-validated (priority 0–1000, visibility timeout 0–43200, wait time 0–20, etc.).
• Delete operations (non-dedup hard delete + dedup tombstone) are transactional — partial deletes cannot occur on crash.
• Queue length checks exclude tombstone records, preventing phantom fullness from blocking sends.
• DLQ moves check destination queue length and fall back to deletion if the DLQ is full.
• Config files reject duplicate keys, unknown keys, and out-of-range values on startup.

---

Comparison with AWS SQS

Feature	SQS	Sasquatch
Visibility timeout	✅	✅
Long polling	✅	✅
Dead letter queue	✅	✅
Batch send (1–10)	✅	✅
Batch receive (1–10)	✅	✅
Batch delete (1–10)	✅	✅
ChangeMessageVisibility	✅	✅
Message delay	✅	✅
Per-message retention	✅ (queue-level)	✅ (per-message)
Content deduplication	✅ (FIFO queues)	✅
Receipt handle	✅	✅
MD5 verification	✅	✅
Message priority	❌	✅
LIFO ordering	❌	✅
Peek (read-only receive)	❌	✅
Shuffle (random receive)	❌	✅
Message attributes	✅	✅ (string key-value)
FIFO exactly-once	✅	❌ (at-least-once)
Queue-level config	✅	❌ (implicit queues)
IAM / auth	✅	❌

---

Scaling Beyond SQLite

If you hit write throughput limits (SQLite serializes all writes through a single connection), the migration path to PostgreSQL is straightforward:

• The SQL patterns translate directly (visible_at <= ? becomes visible_at <= NOW())
• Replace BEGIN IMMEDIATE with SELECT ... FOR UPDATE SKIP LOCKED for non-blocking concurrent receives
• The reader/writer split becomes unnecessary — Postgres handles concurrent writers natively
• The notify channel can be replaced with PostgreSQL's LISTEN/NOTIFY for zero-poll long polling