Microservices Architecture: Building Blocks, Patterns, and Observability on AWS

Microservices sound simple on a whiteboard. Draw some boxes, connect them with arrows, label one "API Gateway" and another "Queue" — done. Ship it. Promote yourself to architect (or so I wished).

In reality, a microservices architecture is a distributed systems problem disguised as an organizational one. Every arrow on that whiteboard is a network call that can fail. Every box is a service that can crash independently. And every boundary you draw is a contract you now have to maintain, version, and monitor.

I've spent enough time debugging cascading failures across services when I should be sleeping to know that the difference between a well-designed microservice system and a distributed monolith isn't the technology — it's the discipline and the trade-offs behind each decision. In this post, I cover the building blocks on AWS, the patterns that keep systems resilient, and the observability practices that let you sleep through the night (most nights).

Microservices Architecture — AWS building blocks and communication patterns visualized

The AWS Building Blocks

Before diving into patterns and principles, let's map the core infrastructure components. Each building block serves a specific role, and choosing the wrong one costs you either money, performance, or portions of your weekend.

Building Block	AWS Service	Role	Best For
API Edge	API Gateway	Public-facing entry point: auth, throttling, rate limiting	Centralized auth and rate limiting before traffic reaches your VPC
Serverless Compute	Lambda	Event-driven, pay-per-invocation functions	Short-lived tasks, event processing, glue logic
Container Compute	ECS on Fargate	Managed containers without managing servers	Long-running HTTP/gRPC microservices
Optimized Compute	ECS on EC2	Containers on instances you control	Cost optimization at scale with Savings Plans
Full Control	EC2	Raw virtual machines	Legacy workloads, custom runtimes, or when you enjoy feeling like a sys admin
Queue	SQS	Durable async buffer with dead-letter queues	Decoupling producers from consumers, retry with backoff
Pub/Sub	SNS	Fan-out messaging to multiple subscribers	Broadcasting events to N downstream services
Event Bus	EventBridge	Schema-aware event routing with filtering and replay	Cross-service events with content-based routing
NoSQL	DynamoDB	Key-value and document store	High-throughput, low-latency reads/writes with minimal ops. Use on-demand capacity for unpredictable traffic; switch to provisioned with auto-scaling once patterns stabilize. For infrequently accessed tables, the Standard-IA table class cuts storage costs by up to 60% — but increases throughput (read/write) costs by ~25%, so evaluate your request-to-storage ratio before switching
Relational	Aurora	Managed MySQL/PostgreSQL with read replicas	Transactional workflows, complex queries, relational integrity. Aurora Serverless v2 scales down to 0.5 ACU for variable-traffic microservices, eliminating capacity planning — though it does not pause to zero at idle (v1 supported this but is deprecated and unavailable for new databases)
Service Networking	VPC Lattice	Cross-service networking, auth, and discovery	When managing multiple services across VPCs with IAM-based service-to-service auth. For teams already on ECS, ECS Service Connect offers simpler service mesh capabilities across clusters sharing a Cloud Map namespace

REST API vs HTTP API: API Gateway offers two flavors. REST APIs (v1) are feature-rich but heavier. HTTP APIs (v2) are designed for low-latency microservices and are up to 71% cheaper ($1.00/million vs $3.50/million requests). Don't default to REST APIs just for auth — HTTP APIs natively support JWT/OIDC authorizers, WAF integration, and X-Ray tracing. Reach for REST APIs when you need features HTTP APIs still lack: request validation, response caching, canary deployments, private endpoints, or API key-based usage plans. AWS has been progressively closing this gap — check the current feature comparison before deciding, as the remaining differentiators continue to shrink.

Choosing Your Compute Layer

This is where I've personally overthought on multiple occasions. The decision tree is simpler than it looks:

Lambda — Your function runs for under 15 minutes, is triggered by events (SQS, S3, API Gateway), and doesn't need persistent connections. The cold start tax is real but manageable for most workloads — and addressable via Provisioned Concurrency (pre-initialized environments you pay for whether invoked or not) or SnapStart (available for Java, .NET, and Python runtimes), which snapshots the initialized environment to slash cold starts without ongoing provisioned cost.
ECS on Fargate — You have "normal" microservices: HTTP APIs, gRPC services, background workers that need to stay warm. You don't want to think about instances. This is the sweet spot for most teams.
ECS on EC2 — You've hit scale where Fargate's per-vCPU pricing hurts, and you can commit to Savings Plans. This is a cost optimization play, not an architectural one.
EC2 — You have legacy software that can't be containerized, or compliance requirements that demand OS-level control for whatever reason (probably regulatory).

The SNS → SQS Pattern

This deserves its own callout because it's the backbone of resilient event-driven architectures on AWS.

SNS alone has retry policies for HTTP/S and Lambda subscribers, but offers no built-in buffering for consumer backpressure — if retries are exhausted and there's no DLQ on the subscription, the message is gone. SQS alone is point-to-point — it supports multiple producers and competing consumers, but each message is delivered to exactly one consumer. Combining them gives you durable fan-out: SNS broadcasts the event, each subscriber has its own SQS queue that buffers messages independently, and failed processing gets routed to a dead-letter queue (DLQ) for inspection and replay.

[Service A] → publishes event → [SNS Topic]
                                      ├── [SQS Queue → Service B]
                                      ├── [SQS Queue → Service C]
                                      └── [SQS Queue → Service D]

Each consumer processes independently, at its own pace, with its own retry policy. Service B being slow doesn't block Service C. Service D being down doesn't lose messages — they wait in the queue.

Treat your events as contracts: version them, validate their schema, and document their semantics. An event named order.updated that sometimes means "order was shipped" and sometimes means "order address changed" isn't an event — it's a recipe for confusion.

Don't forget the DLQ. Every SQS dead-letter queue should have a CloudWatch alarm on ApproximateNumberOfMessagesVisible — if messages are piling up, you need to know immediately, not weeks later during an audit. When the root cause is fixed, use SQS's native DLQ redrive to replay failed messages back into the source queue. For critical workflows, consider attaching a Lambda function to the DLQ that sends an alert or triggers a compensating action automatically. And don't overlook SNS subscription DLQs — Lambda, HTTP/S, and SQS subscriptions can each have their own DLQ attached. Apply the same alarm strategy there.

When to Reach for EventBridge

The SNS → SQS pattern is battle-tested, but EventBridge is worth evaluating for cross-service event routing. It provides content-based filtering (route events by payload fields, not just topic), a schema registry for event discovery and validation, and built-in archive and replay for reprocessing historical events. If your event routing logic is getting complex — multiple consumers filtering for specific subsets of events — EventBridge can replace a tangle of SNS filter policies with cleaner, more expressive rules.

Microservice Best Practices

Principles that sound obvious until you're three sprints deep and realize you've accidentally built a distributed monolith.

Database Per Service

This is the north star of a good microservice design. Every service that requires data owns its. Period.

If Service A needs data from Service B, it asks via an API call or consumes an event. It doesn't reach into Service B's database with a cross-service query. The moment two services share a database, you've coupled their deployment cycles, their schema migrations, and their on-call rotations.

I've lived through the "shared database" approach. It starts innocently — "it's just one cross-database join" — and ends with an X-table migration that requires coordinating three teams and a prayer circle.

The hard part is reads that span services — "show me the order with the customer name and payment status" touches three service boundaries. Patterns like API composition (aggregating responses from multiple services) and CQRS (maintaining read-optimized projections from event streams) address this, but that's a post of its own.

Bounded Contexts

One service = one business capability = one data owner.

An OrderService owns orders. A PaymentService owns payment processing. They communicate through well-defined interfaces. If you find yourself asking "should this logic live in Service A or Service B?" — you probably have a boundary problem, not a code problem.

Resilience Patterns

In a distributed system, failure is not exceptional — it's Monday. Design for it:

Timeouts — Every outbound call gets a timeout. No exceptions. A service that hangs indefinitely because a downstream dependency is slow will exhaust its connection pool, and then you have two broken services instead of one.
Retry with exponential backoff — Retry 1 at 1s, retry 2 at 2s, retry 3 at 4s. Add jitter to prevent thundering herds. Never retry without backoff unless you enjoy turning a partial outage into a full one.
Circuit breakers — After N consecutive failures, stop calling the downstream service entirely. Give it time to recover. Trip → wait → half-open → test → close (or trip again). For long-running ECS containers, in-process libraries like opossum work out of the box. On Lambda, the per-invocation isolation model contains blast radius — a failed invocation doesn't poison other concurrent executions. For SQS-triggered Lambdas, visibility timeouts and DLQs provide natural failure containment without maintaining circuit state. However, for synchronous invocation paths (e.g., API Gateway → Lambda), thousands of concurrent invocations can still hammer a degraded downstream service — consider a shared circuit state check (e.g., a flag in DynamoDB or ElastiCache) or use AWS Step Functions to orchestrate retries, backoff, and failure thresholds across distributed steps.
Bulkheads — Isolate resources per dependency. In ECS/EC2 services, Service A's thread pool for calling the payment service should be separate from its pool for calling the inventory service. One slow dependency shouldn't starve all others. Lambda's per-invocation model provides natural isolation, but you can use reserved concurrency as a function-level bulkhead — capping how many concurrent invocations one function can consume prevents it from starving other functions of the account's concurrency pool.

Idempotency

SQS standard queues deliver messages at least once. Lambda retries failed asynchronous invocations automatically, and SQS event sources reprocess failed messages. HTTP clients retry on timeout. Every network boundary in your system is a place where duplicate processing can happen.

SQS FIFO queues provide exactly-once processing and strict message ordering out of the box. If your workflow requires both — and you can work within the default per-queue throughput ceiling (300 transactions/s across all message groups, or up to 3,000/s when using batch APIs like SendMessageBatch) — FIFO queues eliminate much of the deduplication burden at the infrastructure level. High-Throughput FIFO mode removes that per-queue ceiling by partitioning throughput per message group, supporting up to 6,000 transactions per second per API action (SendMessage, ReceiveMessage, DeleteMessage) per queue — ordering and deduplication remain scoped to the message group ID. For everything else, handle idempotency in application code.

Clients should retry safely by:

Using idempotency keys — Send an Idempotency-Key: <UUID> header with every mutating request. The server stores the key and returns the cached result if it's seen again.
Exponential backoff — As described in the resilience patterns above, use jittered backoff to prevent hammering a recovering server back into failure.

On the server side, don't write custom database locking boilerplate to handle this. If you're on Lambda, leverage the AWS Lambda Powertools Idempotency utility — it handles the DynamoDB storage lifecycle, prevents race conditions, and safely returns cached responses for identical requests out of the box. For non-Lambda services, use a business-level deduplication key (like an order or transaction ID) stored in an atomic cache like ElastiCache — don't rely on the SQS-assigned MessageId for deduplication, as a producer retrying a send after a timeout will cause SQS to assign a new MessageId to the duplicate message.

Security

Least-privilege IAM per service — Each Lambda function or ECS task gets its own IAM role with only the permissions it needs. Not a shared role with * permissions (yes, I've seen it in production. Yes, it was mine, let's not talk about 2020).
Secrets Manager or SSM Parameter Store — API keys, database credentials, and third-party tokens live in Secrets Manager or SSM Parameter Store (SecureString), not in environment variables pasted from a Slack/Teams DM. The key differentiator: Secrets Manager supports automatic credential rotation (Lambda-based rotation for custom integrations, or native rotation for RDS, DocumentDB, and Redshift) — Parameter Store doesn't. Secrets Manager charges per secret and per API call, so for high-invocation Lambda functions, use the Lambda extension caching layer or SDK-level caching to avoid surprise bills. SSM Parameter Store standard parameters are free and work well for static configuration and non-rotated secrets.
Let the API Gateway handle perimeter defense — Offload initial authentication, global rate limiting, and CORS to the edge. This protects your downstream services from resource exhaustion. However, don't let this be your only defense — your application code should still verify origin context headers or enforce IAM resource-based policies to ensure zero-trust security inside the VPC.

Validation and Error Handling

Never Trust the Client

This applies whether "the client" is a browser, a mobile app, or another internal service. Trust nothing that crosses a network boundary:

Syntax validation — Is the payload valid JSON? Does it conform to the expected schema? Reject malformed requests at the edge before they consume any downstream resources.
Semantic validation — Do the values make sense? Is the age greater than zero? Does the referenced product ID actually exist? A syntactically perfect request can still be semantically nonsensical.
Boundary strategy — Use the API Gateway or a validation middleware to reject invalid requests early. Every invalid request that penetrates deeper into your system is wasted compute, wasted logging, and wasted time debugging.

Error Response Discipline

Never leak implementation details. A 500 error should not include a stack trace, a database query, a library version, or an internal service name. Every piece of information you expose in an error response is a gift to anyone probing your system.

4xx errors — Be specific. Tell the client exactly what's wrong so they can fix it: "The quantity field must be greater than zero."
5xx errors — Be generic. The client can't fix your server. "An internal error occurred. Reference: req_abc123."
Bridge the gap — Return a request-id or trace-id in every error response. When the client reports a failure, you can correlate it to your internal logs without exposing those logs to the client.

Logging Errors vs. Returning Them

Return only what the client needs to understand or fix the failure. Log everything else internally — the full stack trace, the request payload (with sensitive fields like passwords, payment info, and PHI/PII redacted), the correlation ID, the downstream service that actually failed.

The error response is a user interface. The error log is a debugging tool. They serve different audiences and should contain different information.

Authentication vs. Authorization

These two get conflated constantly, and the HTTP spec doesn't help (looking at you, 401 Unauthorized that actually means "unauthenticated").

Authentication — Who are you? Prove your identity. (JWT, session token, API key)
Authorization — What can you do? Given your identity, are you allowed to perform this action on this resource?

The Status Codes

401 Unauthenticated — "No offense but, who the hell are you?" No valid credentials were provided, or the token expired.
403 Forbidden — "I know exactly who you are. You still can't touch this." Valid identity, insufficient permissions.

Where Auth Is Enforced

Fail fast, but trust no one. The API Gateway should validate the JWT or session token on every request — this requires explicitly configuring an authorizer (JWT/OIDC for HTTP APIs, or a Cognito/Lambda authorizer for REST APIs). Without one, the gateway passes all traffic through unchecked. With an authorizer in place, invalid requests die at the edge — before they cost any compute inside your VPC, before they hit a Lambda function, before they touch a database.

However, relying only on the gateway violates zero-trust principles. If an internal service or a misconfigured route bypasses your gateway, an unprotected backend will blindly trust the traffic. Your Lambda functions and ECS tasks should still verify the origin context — such as validating a gateway-signed header or enforcing strict IAM resource policies — before executing business logic. The gateway is your first line of defense, not your only one.

For business-level authorization (e.g., "can this user edit this specific order?"), use middleware or a dedicated authorization layer within the service. These checks should be stateless where possible — decode the JWT claims, verify ownership, proceed or reject. If you find yourself making a database call to check permissions on every request, consider caching the authorization decision, restructuring your token claims, or offloading policy evaluation entirely to Amazon Verified Permissions using the Cedar policy language.

Observability

You cannot debug what you cannot see. In a monolith, a request enters and exits one process — you read one log file. In a microservice architecture, a single user action might traverse five services, two queues, and a database. Without proper observability, you're debugging with a blindfold.

The Three Pillars

Metrics — The "what." Numerical measurements over time. CPU utilization, request count, error rate, p99 latency. Metrics are cheap to store, fast to query, and perfect for dashboards and alerts. They tell you that something is wrong.

Traces — The "where." The journey of a single request as it travels across multiple services. A trace shows you that the /checkout endpoint took 4.2 seconds because Service B's database call took 3.8 of those seconds. Traces are essential for finding bottlenecks and understanding dependencies.

Logs — The "why." Immutable, fine-grained evidence. "Database connection timeout at 12:03 AM caused by missing index on orders.customer_id." Logs provide the context that metrics and traces can't: the actual error message, the payload that caused it, the state of the system at that moment.

Correlation IDs

This is the glue that makes the three pillars stick together.

When a request hits the API Gateway, generate a unique UUID and attach it as a header (Correlation-Id). Every service that handles the request must:

Capture the correlation ID from the incoming event or HTTP header.
Propagate it to every outbound call (HTTP requests, SQS messages, SNS publications).
Log it in every log entry using structured logging.

When something breaks at 3 AM, you take the correlation ID from the alert, search across all services, and reconstruct the full journey of the failed request. Without correlation IDs, you're grep-ing through 12 log streams trying to match timestamps. With them, you run one query and see everything.

Tooling on AWS:

AWS Lambda Powertools — Automatically captures and propagates correlation IDs, adds structured logging, and integrates with X-Ray out of the box.
AWS CloudWatch Application Signals & X-Ray — Distributed tracing across Lambda, API Gateway (both REST and HTTP APIs), SNS, SQS, and DynamoDB. This links your traces directly to your structured CloudWatch logs, showing you the service map, latency bottlenecks, and application errors in a single pane of glass. For production services, configure X-Ray sampling rules to manage cost — trace 100% of errors and a small percentage (e.g., 1–5%) of successful requests rather than tracing everything.

If your observability needs outgrow CloudWatch — advanced anomaly detection, cross-account correlation at scale, or unified dashboards across AWS and non-AWS infrastructure — platforms like Datadog or Splunk integrate natively with AWS via CloudWatch metric streams, S3 log exports, and X-Ray trace forwarding. They're not required to get started, but worth evaluating once operational complexity justifies the cost.

Monitoring vs. Observability

These aren't synonyms:

Monitoring is for known unknowns — things you know can break, so you build dashboards for them. CPU above 90%, disk above 80%, error rate above 1%. You've anticipated these failure modes and set up alerts.
Observability is for unknown unknowns — things you couldn't predict. A new edge case, a race condition that only manifests under specific load patterns, a third-party API returning unexpected data. Observability means you have enough raw signals (metrics, traces, logs) to ask questions you didn't know you'd need to ask.

Monitoring tells you the house is on fire. Observability helps you figure out why it started in the guest bathroom.

Putting It Together

Microservices aren't a goal — they're yet another trade-off. You trade the simplicity of a monolith for independent deployability, team autonomy, and the ability to scale components independently. But that trade only pays off if you invest in the supporting infrastructure: resilience patterns, proper boundaries, idempotent operations, security at every layer, and observability that lets you understand what's actually happening in production.

The building blocks on AWS are mature and well-documented. The hard part was never "which service do I use?" — it's "how do I design the boundaries, handle failures gracefully, and maintain visibility across dozens of independently deployed services?"

Get the fundamentals right — DB per service, fail fast at the edge, retry with backoff, correlate everything — and the architecture scales with your team. Skip them, and you'll end up where every under-invested microservices team ends up: a distributed monolith with all the complexity of both approaches and the benefits of neither.

And if you're starting greenfield — consider a modular monolith with clean internal boundaries first. Extract services only when deployment coupling or independent scaling demands justify the distributed complexity. The best microservice extraction is one where the boundary was already clean before you drew the network line.