"Design a highly available web app on AWS" is the most common cloud design question in DevOps interviews, and the follow-ups are precise: why a NAT gateway per zone, why a role instead of an access key, what an explicit deny does, how target tracking picks a number. In real work the same facts decide whether a release is safe (instance refresh, health checks), whether the bill is sane (NAT data processing, cross-zone transfer) and whether an audit passes (no long-lived keys, IMDSv2, block public access). Kubernetes on EKS sits on top of all of this, so the next lesson assumes it.

Accounts, regions and the shared responsibility model

An AWS account is the hard boundary for permissions, billing and blast radius. Put production in its own account, separate from dev and staging, and group accounts with AWS Organizations: a management account, organisational units (OUs) and service control policies (SCPs). An SCP sets the maximum permissions available in an account; it never grants anything. A classic SCP denies every region except the ones you use, so a leaked key cannot start miners elsewhere. One account per environment means a wrong terraform destroy in staging cannot touch production data.

A region (ap-south-1 is Mumbai) is a set of Availability Zones (AZs): separate data centres with separate power and network, close enough for synchronous replication. Everything here spreads across three AZs because losing one AZ is the failure AWS designs for.

The shared responsibility model: AWS is responsible for security of the cloud (hardware, regions, the hypervisor, the managed service software). You are responsible for security in the cloud: your data, IAM, OS patches on EC2, security groups and encryption settings. "RDS is managed" does not mean AWS chose your password or backup window.

IAM: identities, policies and how a decision is made

• Users are long-lived identities with a password or access keys. Avoid them. Humans sign in through IAM Identity Center (formerly AWS SSO) with the company identity provider and get short-lived credentials for a permission set in an account; aws configure sso sets this up for the CLI.
• Roles are identities that are assumed, not logged into. An EC2 instance assumes its instance role through the metadata service, a CI job through OIDC, a Lambda function through its execution role. The credentials are temporary and rotate by themselves, so there is nothing to leak.
• Identity-based policies attach to users, groups and roles and say what that identity may do. Resource-based policies attach to a resource (an S3 bucket policy, a KMS key policy, a role's trust policy) and say who may use it.

The evaluation logic, in the order AWS applies it. Everything starts as an implicit deny. Then:

1. An explicit Deny in any applicable policy (SCP, resource policy, identity policy, permissions boundary, session policy) ends it as Deny. Nothing overrides an explicit deny.
2. The organisation's SCPs (and resource control policies, if used) must allow the action.
3. A resource-based policy that allows the principal can be enough on its own within the same account.
4. Otherwise an identity-based policy must allow it.
5. A permissions boundary on the identity, and a session policy passed at assume time, must each allow it too.

The interview version: "explicit deny wins, then the request needs an allow from every layer that applies, and the default is deny."

Least privilege is a practice, not a flag. Start from an AWS managed policy, let IAM Access Analyzer generate a policy from what the identity actually used, scope Resource to ARNs, and add conditions such as aws:RequestedRegion or s3:prefix.

CI through OIDC, no keys. You create an OIDC identity provider in IAM for https://token.actions.githubusercontent.com with audience sts.amazonaws.com. The role's trust policy allows sts:AssumeRoleWithWebIdentity only for a sub claim such as repo:my-org/my-repo:ref:refs/heads/main. The workflow sets permissions: id-token: write and calls aws-actions/configure-aws-credentials with role-to-assume. No secret is stored anywhere.

VPC: the network you own

A VPC is a private network in one region with an IPv4 range from /16 to /28. Pick a /16 (65,536 addresses) that does not overlap any VPC or office network you may ever peer with. A subnet is a slice of it in one AZ; AWS reserves the first four and the last address of every subnet, so a /24 has 251 usable addresses. Three tiers per AZ:

A VPC across three Availability Zones with public subnets holding ALB nodes and NAT gateways, private subnets holding Auto Scaling group instances, and database subnets holding an RDS primary and its synchronous standby

• Public subnets have a route table whose default route (0.0.0.0/0) points at the internet gateway (one per VPC). Only load balancer nodes and NAT gateways live here.
• Private subnets have a default route to a NAT gateway, so instances can reach the internet (package updates, third-party APIs) without being reachable from it.
• Database subnets have no default route at all.

A NAT gateway is zonal. If every private subnet routes to the NAT in zone a and zone a fails, zones b and c lose internet access too, so the documented pattern is one NAT gateway per AZ with each private route table pointing at its own zone's NAT. NAT gateways are billed per hour and per GB processed (an assumed 0.045 USD per hour and 0.045 USD per GB), which adds up when nodes pull images through them. Two cheap fixes: a gateway VPC endpoint for S3 (free, one route table entry) and interface endpoints for ECR (ecr.api and ecr.dkr, billed hourly). Image layers live in S3, so ECR needs the S3 gateway endpoint as well.

Security groups are stateful firewalls on network interfaces: allow rules only, return traffic allowed automatically, and a rule can reference another security group ("allow 8080 from the ALB's security group") so you never hard-code IPs. Network ACLs are stateless rules on a subnet, evaluated in number order, with allow and deny; being stateless, they need the ephemeral return ports opened by hand. Use security groups for almost everything and NACLs as a coarse second layer, such as denying a known bad range at the subnet edge.

VPC peering joins two VPCs one-to-one with non-overlapping CIDRs and is not transitive; Transit Gateway is the hub for many VPCs or an on-premises link.

EC2: the machines

An instance type name encodes family, generation and options: m6i.xlarge is general purpose (m), generation 6, Intel (i), size xlarge (4 vCPU, 16 GiB). c is compute optimised, r memory optimised, t burstable (fine for dev, a trap for steady load because it runs out of CPU credits); after the generation, g means Graviton (Arm), a AMD, d local NVMe disk, n more network. An AMI is the disk image an instance boots from; build your own with Packer, or take Amazon Linux 2023 and configure it with user data, a script run once on first boot as root.

The instance metadata service at 169.254.169.254 is how a process on the instance learns its own ID, region and, most importantly, the temporary credentials of the instance role. IMDSv2 requires a session token from a PUT request, and the default hop limit of 1 stops a container or an SSRF through a proxy from reaching it. Always set HttpTokens=required in the launch template.

# IMDSv2: get a token (TTL up to 21600 s), then use it
TOKEN=$(curl -sX PUT 169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" 169.254.169.254/latest/meta-data/instance-id
# require IMDSv2 on an existing instance
aws ec2 modify-instance-metadata-options --instance-id i-0123456789abcdef0 --http-tokens required

EBS volumes are network-attached disks in one AZ. Use gp3: 3,000 IOPS and 125 MiB/s are included and you can raise either independently of size (gp2 ties IOPS to size, 3 per GiB, and bursts from credits). Snapshots are incremental copies to S3 and the basis of AMIs and backups. Placement groups: cluster packs instances for low latency, spread puts each on separate hardware (up to 7 per AZ), partition splits a fleet into groups that share no racks.

Auto Scaling groups

An Auto Scaling group (ASG) keeps a fleet of instances built from a launch template (AMI, instance type, security groups, instance role, user data, IMDSv2 setting) across the subnets you give it, with min, desired and max counts, replacing instances that fail health checks. EC2 status checks are on by default; turn on ELB health checks too so an instance whose app is dead but whose kernel is fine is replaced. The health check grace period (300 s in the console, 0 through the CLI) stops a slow-booting instance being killed before it is up; the default instance warmup keeps a new instance's CPU out of the group average until it settles.

• Target tracking is the one to use. You name a metric and a target (ASGAverageCPUUtilization at 60, or ALBRequestCountPerTarget at a number you measured); AWS creates the CloudWatch alarms and computes the capacity. The metric must move in proportion to instance count (CPU per instance does; total request count does not). Scale-out is fast; scale-in is deliberately slow and gradual.
• Step scaling adds N instances when an alarm crosses a threshold; simple scaling is the old form with a cooldown (default 300 s). Scheduled actions handle the 9 am ramp you know is coming.

# target tracking: hold average CPU near 60 percent
aws autoscaling put-scaling-policy --auto-scaling-group-name web-asg \
  --policy-name cpu60 --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{"TargetValue": 60.0,
    "PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"}}'

Lifecycle hooks pause an instance in Pending:Wait (configure before it serves) or Terminating:Wait (drain, ship logs) until you call complete-lifecycle-action or the heartbeat timeout (default 1 hour) expires. Instance refresh rolls a new launch template version through the group in batches, keeping at least MinHealthyPercentage (default 90) of desired capacity in service and growing to at most MaxHealthyPercentage (default 100); set 100 and above to launch before terminating, and turn on AutoRollback so failed health checks revert it. Vertical scaling (a bigger instance) is a restart with a ceiling; horizontal scaling (more instances behind a load balancer) is what an ASG gives you. Building one yourself is the week 29 project: a loop that reads a metric, computes a desired count, and starts or stops VMs.

Load balancers

A target group holds the targets (instances, IP addresses, or a Lambda) and the health check: HTTP GET on a path every 30 s, timeout 5 s, healthy after 5 passes, unhealthy after 2 failures, success on 200 by default. If every target is unhealthy the ALB fails open and sends traffic to all of them. Deregistration delay (default 300 s) lets in-flight requests finish before removal. Sticky sessions pin a client to one target with a cookie; session state in Redis is the better fix. An ALB needs subnets in at least two AZs, each /27 or larger with eight free addresses.

S3

An S3 bucket has a globally unique name and lives in one region. Objects have keys; "folders" are prefixes. Turn on versioning (overwrites and deletes become recoverable) and add lifecycle rules: logs to STANDARD_IA after 30 days, to GLACIER_IR or DEEP_ARCHIVE after 90, expire after 400, abort incomplete multipart uploads after 7 days. Intelligent-Tiering moves objects between tiers for a small fee. Keep Block Public Access on at the account level; a public website goes behind CloudFront with origin access control. A presigned URL lets a client upload or download one object for a limited time with the signer's permissions: up to 7 days with IAM user keys, but one signed with a role's temporary credentials dies when those credentials expire, the usual surprise.

RDS and Aurora

RDS runs PostgreSQL, MySQL and others on an instance you size. Multi-AZ keeps a synchronous standby in another AZ; it serves no reads, and on failure RDS flips the DNS record to the standby in typically 60 to 120 seconds, so the app must reconnect and must not cache DNS forever. A read replica is asynchronous and serves reads: scaling, not failover. Backups: daily snapshots plus transaction logs for point-in-time restore within the retention window. A parameter group is the engine config (max_connections, shared_buffers, log_min_duration_statement); the default max_connections is derived from instance memory, which is why a small instance falls over at a few hundred connections and why PgBouncer or RDS Proxy sits in front of it.

Aurora separates compute from a storage layer replicated across three AZs; up to 15 replicas share that storage, failover promotes a replica fast, and there is a writer endpoint and a reader endpoint (which points at the writer if there are no replicas). For the profile this site targets (many services, one shared relational database, strict availability), Aurora PostgreSQL is the default answer: three-AZ durability without an idle standby, fast failover, and read scaling through the reader endpoint.

Route 53, CloudWatch, Systems Manager, Secrets Manager

Route 53 hosts a hosted zone per domain. An alias record points a name, even the zone apex example.com, at an AWS resource (ALB, CloudFront, S3 website) and is free to query; a CNAME cannot sit at the apex. Routing policies: simple, weighted (5 percent to a new stack), latency, failover (health check on the primary), geolocation, multivalue.

CloudWatch holds metrics (EC2 sends CPU, network and disk every 5 minutes; detailed monitoring makes it 1 minute for a charge; memory and disk usage need the CloudWatch agent inside the instance), logs (log groups with retention, shipped by the agent), alarms (a metric over a threshold for N periods triggers SNS, an ASG policy or EventBridge) and dashboards.

Systems Manager Session Manager gives a shell on an instance through the SSM agent and IAM, with no inbound port 22, no bastion and no key pairs, and sessions can be logged to S3 or CloudWatch: aws ssm start-session --target i-0123456789abcdef0. Parameter Store holds configuration as a path hierarchy (/prod/api/db_host), SecureString encrypted with KMS, standard tier free up to 4 KB per value. Secrets Manager is for credentials: it rotates them with a Lambda function, replicates across regions, and costs per secret per month, so non-secret config stays in Parameter Store.

The CLI and profiles. ~/.aws/config holds [profile name] sections with region, output and the SSO session; ~/.aws/credentials holds [name] sections with keys and should be nearly empty on a modern laptop. --profile prod or AWS_PROFILE=prod selects one; run aws sts get-caller-identity before anything destructive.

The one setting people get wrong

Subnet plan. VPC 10.20.0.0/16 in ap-south-1, three AZs, three tiers. Private subnets carry the instances and later the EKS pods, so they get the big blocks: 10.20.0.0/20, 10.20.16.0/20, 10.20.32.0/20, each 4,096 addresses, 4,091 usable after the five AWS reserves. Public subnets hold only ALB nodes and a NAT gateway: 10.20.100.0/24, 10.20.101.0/24, 10.20.102.0/24 (251 usable each, well above the eight free addresses an ALB needs per subnet). Database subnets: 10.20.110.0/24 to 10.20.112.0/24. Total used: 3 x 4,096 + 6 x 256 = 13,824 of 65,536, leaving 51,712 addresses for pod subnets or a second cluster. None of the nine blocks overlap, and the /16 does not overlap the office range 192.168.0.0/16, so peering later is possible.

ASG sizing. Peak is 2,000 requests per second. A load test shows one m6i.xlarge handles 300 requests per second before p99 latency climbs. Raw need: 2,000 / 300 = 6.67, so 7 instances. Add 30 percent headroom so a traffic spike or a slow deploy does not push instances to their limit: 2,000 x 1.3 / 300 = 8.67, so 9 instances, 3 per AZ. That is desired = 9.

Express the same thing as a target-tracking target: 300 / 1.3 = 230 requests per target. With ALBRequestCountPerTarget at 230, the policy computes 2,000 / 230 = 8.7 and rounds up to 9, the same number, and at night with 400 requests per second it computes 400 / 230 = 1.7, rounded up to 2, which the min = 3 below holds at 3.

Set min = 3 (one per AZ, so the service is never in one zone) and max = 20 (about twice the peak need, a ceiling against a runaway loop).

Now the AZ-loss test. Lose one zone and 6 instances remain: 6 x 300 = 1,800 requests per second, 200 short of the peak. Two choices: accept that the ASG will add instances in the surviving zones within about 5 minutes (launch plus warmup), or set desired = 15 (5 per AZ) so any two zones carry 10 x 300 = 3,000. At an assumed 0.192 USD per hour, 9 instances cost 9 x 0.192 x 730 = 1,261 USD per month and 15 cost 2,102 USD; the difference, 841 USD, is the price of surviving a zone loss at peak without waiting for a scale-out. Most teams choose 9 and a fast scale-out; a payment system chooses 15.