Cloud+ Certification Prep

Fifteen pages that define the exam

CompTIA's objectives document for exam CV0-004 runs to fifteen pages. It is labelled Exam Objectives Version 2.0, carries a 2023 copyright, and lists thirty-three numbered objectives in six domains, each objective followed by the technologies, terms and tasks it includes. A note near the front warns that those lists are examples rather than complete inventories, so a question may touch something an objective implies without naming. Treat the document as the exam's syllabus. This course follows it from objective 1.1 to objective 6.3, and every lesson says near the top which objective numbers it covers. This lesson is the orientation: it covers no objective of its own, but it prepares you for all six domains and builds the lab you will use in every lesson after it.

Here is the test as CompTIA's Cloud+ page and the objectives document described it on 27 October 2026.

Two rows deserve a second look. A passing score of 750 is a point on CompTIA's 100 to 900 scale. It is not a percentage, and nothing on CompTIA's pages converts it into a number of correct answers, so ignore anyone who tells you that you need exactly 83 percent. And the experience row is labelled recommended: CompTIA presents it as the background that makes the exam reasonable, not as a prerequisite you must prove. If subnets, routing tables, DNS and VPNs are still shaky for you, work through Network+ Certification Prep (CS 215) first or alongside this course, because Cloud+ builds on all of them.

Where the marks are

CompTIA publishes the weight of each domain, the share of the exam drawn from it. The weights are uneven, and so are the objective counts behind them.

Read the table in both directions. Cloud Architecture is the heaviest domain, but its 23 percent is spread across eleven objectives, about 2 percent each. Troubleshooting's 12 percent rests on only three objectives, 4 percent each, and Operations gives about the same per objective. If CompTIA applied the weights directly to a full 90-question form, architecture would supply about 21 questions and each troubleshooting objective three or four. CompTIA publishes percentages, not question counts, so use that arithmetic to plan your effort, not to predict your exam.

Reading an objective by its first words

Every objective opens with a phrase that tells you what kind of knowing the exam expects. Counted across all thirty-three:

Almost half the objectives, then, put you in a situation and ask you to act. Expect a scenario objective such as 3.2, on scaling, to arrive as a situation rather than a definition: a database server that runs short of memory every month-end, and a choice of what to change. That is why this course's quizzes are short scenarios and why every lesson has a lab.

Key idea: a domain's weight tells you how much of the exam it supplies; an objective's opening words tell you how it will be asked. Scenario objectives need practice with your hands, not just definitions.

Simulations, the clock and the order you work in

Some questions are performance-based questions, PBQs, in which you act rather than choose: put steps in order, match settings to requirements, or complete a configuration in a simulated console. CompTIA's explainer on PBQs separates two kinds. Simulations imitate an environment or tool with restricted functions, fill the testing area and have a reset button. Virtual environments run real software and have no reset. The same page lists Cloud+ among the certifications that use simulations only, says you can skip a simulation and return to it later, and says partial credit may be given.

Ninety minutes for up to ninety questions is a minute each on average, and a simulation takes longer than a multiple-choice item. One workable plan is a first pass through the multiple-choice items, flagging the simulations, then a second pass that spends the remaining time on them. Whatever plan you choose, decide it before exam day and rehearse it on timed practice, because a plan improvised under pressure is rarely a good one.

Which version you will sit

CompTIA usually retires an exam about three years after launch, and its Cloud+ page estimates 2027 for CV0-004. A successor is already visible. On 27 October 2026 CompTIA's page of exam objectives under development carried a draft for Cloud+ V5, exam code CV0-005, as draft document version 1.2. Its test details, from question count to passing score, read TBD, and no launch date has been announced. The draft also reshapes the exam into six differently named domains: architecture and design; life-cycle management; operations; security; governance, risk and compliance; and artificial intelligence. Troubleshooting stops being a domain of its own and appears inside the life-cycle, operations and security domains, and no domain is called DevOps.

This course teaches CV0-004, because that is the exam a learner starting now can book and sit. Two habits protect you. Before you buy a voucher, open the Cloud+ page and check which version is on sale and whether a retirement date has been fixed. If CompTIA announces a CV0-005 launch before your planned date, put the two documents side by side and compare them objective by objective. The draft keeps objectives on architecture, provisioning, observability, optimization and security, so most of this course would carry over; its governance and AI domains would need extra study.

One sentence of plain dealing belongs here. CompTIA and Cloud+ are trademarks of CompTIA, and CompTIA does not endorse or sponsor this course. The objectives are paraphrased in the course's own words, and the official document is linked in the sources so you can always check the exact wording. No course can promise that you will pass; this one makes sure nothing on the objectives list is new to you on exam day.

Policies worth reading before you book

• Retakes. CompTIA's retake policy sets no waiting period between a first and a second attempt. Before a third or any later attempt you must wait at least fourteen calendar days after the previous one. Each attempt is a full-price exam.
• Identification. You need two original, unexpired IDs: a government-issued primary ID with your name and a recent photograph, and a secondary ID with your name and either a signature or a photograph.
• Candidates aged 17 and under. Since 23 June 2026 every minor needs consent from a parent or legal guardian, given through CompTIA Central, before registering for an exam, and needs it for each testing event.
• Validity. CompTIA's renewal policy lists Cloud+ in its continuing education program. The certification is valid for three years from the date you earn it and can be renewed without retaking the exam.

A cloud lab with no cloud bill

Every lab in this course runs on your own computer with free tools. A real provider account would show you a console, but it needs an account, sign-up rules differ by provider and country, and a resource you forget to delete keeps costing money. Most of what the exam tests can be practised locally, because the cloud is built from the same parts you can run on a laptop: virtual machines, containers, software-defined networks and automation. Where a lab cannot reproduce a provider's service, the lesson says so and sends you to that provider's public documentation instead.

One account appears in the whole course, and it is optional: a free GitHub account for the pull-request and GitHub Actions extension in Lesson 31. GitHub's terms require users to be 13 or older, and a country's own minimum age may be higher. A free account needs no payment information. Everything else is installed and run locally, and you never scan or probe a system you do not own.

Procedure: build the lab one layer at a time

Install in this order and prove each layer works before adding the next. When something fails later, you will know which layer to suspect.

1. A Linux shell. On Windows 10 version 2004 or later, or on Windows 11, open PowerShell as administrator, run wsl --install and restart. Microsoft's instructions say this enables the features WSL needs and installs Ubuntu; the first time Ubuntu starts, it asks you for a Linux username and password, which exist only on your computer. Run wsl -l -v in PowerShell and check that the VERSION column reads 2. On macOS or Linux, your existing terminal is enough.
2. A container engine. Install Docker Desktop on Windows or macOS, Docker Engine on Linux (including inside WSL), or Podman. Docker Desktop's license makes it free for personal use, education, non-commercial open-source projects and businesses with fewer than 250 employees and less than $10 million in annual revenue; larger organizations need a paid subscription, and the license notes that open-source projects such as Docker Engine keep their own terms. Docker's Windows requirements include a 64-bit processor with second-level address translation, 8 GB of system RAM, and hardware virtualization turned on in the BIOS or UEFI. Prove the layer with docker run hello-world, or podman run hello-world: the engine downloads a tiny test image, runs it as a container, prints a confirmation message and exits.
3. A Kubernetes cluster. Install kind and kubectl: winget install Kubernetes.kind on Windows or brew install kind on macOS, with kubectl from the Kubernetes documentation's install page. Run kind create cluster --name lab. kind's documentation describes it as a tool that runs a local Kubernetes cluster using containers as nodes. Prove the layer with kubectl cluster-info --context kind-lab and then kubectl get nodes; the node named lab-control-plane should report Ready.
4. Look underneath. Run docker ps. One running container is named lab-control-plane: your Kubernetes node is a container. On Windows and macOS that container runs inside a Linux virtual machine, and the virtual machine runs on your physical computer. The pods you start in later lessons will run inside the node container, so the stack is four layers deep, which is the cloud's own trick in miniature: every layer looks like real hardware to the one above it.
5. Version control. Install Git and run git --version. Set your identity with git config --global user.name and git config --global user.email, followed by your name and an email address. Make a folder called cloudplus-lab, move into it and run git init. This repository is your lab notebook: every file you write in later lessons goes in it.
6. Tidy up. kind delete cluster --name lab removes the cluster. You will create clusters again whenever a lesson needs one, which is itself a cloud habit: build from a definition, use, destroy.

When a step fails: a permission nobody granted

Now change one input and watch the result flip. Suppose you chose Docker Engine inside WSL or on a Linux computer, installed it, and ran docker run hello-world as your ordinary user. Instead of the confirmation you get an error saying permission was denied while connecting to the Docker daemon's socket. Put sudo in front of the same command and it works.

Trace it. The docker command is only a client; it sends each request to the Docker daemon through a Unix socket. Docker's post-installation guide explains that by default the root user owns that socket, so other users can reach it only through sudo. The engine is fine, the image is fine and the network is fine; the failure is an authorization decision. Docker's documented fix is to create a docker group if it does not exist with sudo groupadd docker, add yourself with sudo usermod -aG docker $USER, and then log out and back in, or run newgrp docker, so the new membership takes effect. The same guide warns that the docker group grants root-level privileges, so on a shared machine that convenience is a security decision, not a formality.

If you use Docker Desktop and a docker or kind command fails because it cannot reach the engine, check first that the Desktop application is running: the client has nothing to talk to until it is. Notice the method in both cases, because Module 10 builds on it. Reproduce the failure, find what differs between the working and failing runs (here, only who ran the command), locate the layer that made the decision, and read that layer's documentation before reaching for a fix.

How to study so it lasts

In 2006 Nicholas Cepeda and four colleagues pooled 839 assessments of distributed practice from 317 experiments reported in 184 articles. Their sharpest finding concerns timing: the gap between study sessions that produced the best retention grew as the delay before the final test grew. Spacing has to match the horizon. If your exam is three months away, do not stop at revisiting a lesson's quiz the next evening; come back to it after a few days, again after a couple of weeks, and again in your final month.

John Dunlosky and colleagues' 2013 review rated ten common study techniques. Practice testing and distributed practice were the two rated high utility; interleaved practice, mixing different kinds of problems in one session, was rated moderate; rereading and highlighting were rated low. For Cloud+ that means three habits. Answer each lesson's ten questions before you reread anything. Explain to yourself why each wrong option is wrong, because each one is built from a real misconception. And once you are a few modules in, mix questions from different domains in one sitting, since recognizing what kind of problem you face is part of the skill.

Plan on six to eight hours a week; at that pace the course's 35 lessons and labs fill roughly three months. Keep the cloudplus-lab repository as your notebook. Commit each lab's commands with a sentence on what each output meant: writing the meaning down is itself retrieval practice, and the commit history becomes a record of how far you have come.

Key idea: space your reviews to match how far away the exam is, test yourself before rereading, and mix domains once you can.

Common misconceptions

• "Cloud+ is an exam about one provider." The objectives describe concepts and tasks. The only products they name are tools that work across providers: Terraform, Ansible, Docker, Kubernetes, Git, Jenkins, GitHub Actions, Grafana and the ELK stack. This course names AWS, Azure and Google Cloud services only to show you one idea under three labels.
• "You need a cloud account and a payment card to practise." Every lab here is free and local, and provider documentation is public, so you can read it without signing in.
• "750 means I need 83 percent." 750 is a point on a 100 to 900 scale, and CompTIA does not present it as a percentage of correct answers.
• "A performance-based question must be finished before you can move on." That describes virtual-environment PBQs, which Cloud+ does not use. Cloud+ simulations can be skipped and revisited.

What to carry forward

• CV0-004: up to 90 questions in 90 minutes, multiple choice plus simulation PBQs, 750 needed on a 100 to 900 scale.
• Six domains and 33 objectives: Cloud Architecture 23, Deployment 19, Operations 17, Security 19, DevOps Fundamentals 10 and Troubleshooting 12 percent.
• Fifteen objectives open with Given a scenario or Given a set of requirements; practise those with your hands.
• A CV0-005 draft exists with no launch date. Prepare for CV0-004 and recheck CompTIA's Cloud+ page before you book.
• Your lab is a Linux shell, a container engine, kind and kubectl, and Git, each proved before the next. The docker group fix is the troubleshooting method in small: find what differs, find the deciding layer, read its documentation.
• Study by spaced self-testing and mixed practice rather than rereading.

Sources

1. CompTIA. (n.d.). CompTIA Cloud+ (V4) certification: Exam details. Retrieved 27 October 2026. comptia.org
2. CompTIA. (2023). CompTIA Cloud+ CV0-004 certification exam: Exam objectives (Version 2.0). CompTIA (PDF)
3. CompTIA. (n.d.). CompTIA exam objectives under development. Retrieved 27 October 2026. comptia.org
4. CompTIA. (2026). CompTIA Cloud+ CV0-005 V5 certification exam: Draft exam objectives (Draft document version 1.2). CompTIA (PDF)
5. CompTIA. (n.d.). Performance-based questions explained. Retrieved 27 October 2026. comptia.org
6. CompTIA. (n.d.). CompTIA certification retake policy. Retrieved 27 October 2026. comptia.org
7. CompTIA. (n.d.). Candidate ID policy. Retrieved 27 October 2026. comptia.org
8. CompTIA. (n.d.). Certification renewal policy. Retrieved 27 October 2026. comptia.org
9. Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., and Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380. doi.org
10. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., and Willingham, D. T. (2013). Improving students' learning with effective learning techniques. Psychological Science in the Public Interest, 14(1), 4-58. doi.org
11. Microsoft. (2025). Install WSL. Microsoft Learn. learn.microsoft.com
12. Docker. (n.d.). Docker Desktop license agreement and Install Docker Desktop on Windows. Docker Docs. docs.docker.com (license); docs.docker.com (Windows)
13. Docker. (n.d.). Linux post-installation steps for Docker Engine. Docker Docs. docs.docker.com
14. Kubernetes SIGs. (n.d.). kind: Quick start. kind.sigs.k8s.io
15. GitHub. (2026). GitHub terms of service (effective 27 April 2026), section B, Account terms. docs.github.com

Seven pages from NIST

In September 2011 the US National Institute of Standards and Technology published Special Publication 800-145, The NIST Definition of Cloud Computing. Written by Peter Mell and Timothy Grance, it runs to seven pages, and much of the vocabulary on the Cloud+ exam starts there. Cloud computing, it says, is "a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources" that can be provisioned and released quickly with minimal management effort or provider interaction. It then names five essential characteristics, three service models and four deployment models. This lesson covers the service models and the responsibility that comes with each, which is CV0-004 objective 1.1. Lesson 3 takes the deployment models.

The five characteristics are worth knowing by name, because scenario questions use them as clues.

The stack, and who runs each layer

Every computing service sits on the same stack of layers: a building, physical machines, a virtualization layer, an operating system and a runtime that executes code. Above them come the application and, finally, the data and the people allowed to touch it. A service model is simply an agreement about where the provider stops and you begin. Read this table from left to right and watch the provider's share grow.

NIST's own definitions draw the same lines in words.

• Infrastructure as a service. You provision processing, storage, networks and other fundamental resources, and you can run arbitrary software, including operating systems. You do not manage the underlying infrastructure, but you control the operating systems, storage and deployed applications, and possibly some networking components such as host firewalls. A virtual machine is the classic example.
• Platform as a service. You deploy applications built with languages, libraries and tools the provider supports. You control the application and possibly some settings of its hosting environment, but not the network, servers, operating systems or storage beneath it. AWS's documentation for Elastic Beanstalk puts it from the user's side: you build and deploy your application, and the service provisions instances, configures load balancing, sets up health monitoring and scales the environment.
• Software as a service. You use the provider's application through a browser or a program interface. NIST says you control not even individual application capabilities, apart from limited user-specific settings. Microsoft names Microsoft 365 and Dynamics 365 as its SaaS examples; web-based email is NIST's.
• Function as a service. NIST's 2011 list stops at three, and CompTIA's objective adds this fourth. You write a function, a handler, and attach it to a trigger: an HTTP request, a file arriving in storage, a message on a queue. The provider runs the function once per event, scales it with demand and bills per request and execution time. AWS describes Lambda as letting you run code without provisioning or managing servers, with the service handling server maintenance, capacity provisioning, scaling and patching.

FaaS is best understood as the far end of PaaS rather than a separate world. Microsoft's own shared responsibility page lists Azure Functions among its PaaS examples. What FaaS adds is the shape of the unit you deploy: not an application that runs all day, but a short piece of code that runs when something happens. Lambda's documentation says it is designed for short-lived tasks that do not retain or rely on state between invocations. A standard function can run for at most 15 minutes per invocation, with memory from 128 MB to 10,240 MB, and Lambda allocates CPU in proportion to memory, reaching the equivalent of one vCPU at 1,769 MB.

The point: each step to the right hands one more layer to the provider. You trade control for less work, and the trade is a good one only if you did not need the control you gave up.

Four workloads, four models

Comparison becomes useful when you run real requirements through it. Here are four workloads and the reasoning that sorts them.

1. A twelve-person accounting firm needs email, calendars and shared documents by next week, and has no IT staff. Nothing here is custom, and nobody is available to patch or monitor anything. SaaS fits: the firm configures users and sharing settings and the provider runs everything else.
2. A development team has a Python web API. It wants to push new versions several times a day, have the platform scale it, and never patch an operating system. The code is custom, so SaaS is out; the team wants no operating system work, so IaaS is the wrong trade. PaaS fits: Elastic Beanstalk, App Service or App Engine run the platform, and the team owns the code and its settings.
3. A vendor's inventory system needs a particular operating system release, a kernel driver for a barcode scanner gateway, and the security team's endpoint agent. Each of those requirements reaches below the application, into layers that PaaS and FaaS take away from you. IaaS fits, and with it comes the duty to patch and harden the guest operating system.
4. Every photo uploaded to object storage needs a thumbnail. Uploads arrive a few thousand times a day, unevenly, and each resize takes under a second. The work is short, triggered by an event and holds no state between runs. FaaS fits: nothing runs and nothing is billed between uploads.

Change one requirement and the answer flips

Now alter the fourth workload. The same upload event must transcode a video, and each transcode takes about forty minutes. Nothing about the trigger changed, but the job now outlasts the 15-minute ceiling on a standard Lambda invocation, and raising the timeout cannot help, because 900 seconds is the maximum. The better design keeps the event and changes the compute: the upload event starts a container task or a batch job on virtual machines, which can run as long as the work needs. A requirement for state can flip the answer the same way. A function that must remember something between runs has to keep it in a database or a storage service, because the platform does not promise that anything survives from one invocation to the next.

The third workload flips too, in the other direction, if the vendor ships a container image that bundles its own driver and needs no host changes. Then a platform that runs containers may serve where only a virtual machine would before. Service model choices are rarely about taste; they follow from which layer a requirement touches.

Where the line falls at three providers

Choosing a model also chooses a division of security work, which is what the shared responsibility model describes. All three major providers publish one, in different words.

AWS splits the work into security of the cloud, which is AWS's job, and security in the cloud, which is the customer's. For an infrastructure service such as Amazon EC2, AWS's page says the customer manages the guest operating system, including updates and security patches. The customer also manages any software installed on the instance and the security group firewall configured on each instance. For more abstracted services such as Amazon S3 and DynamoDB, AWS operates the infrastructure, operating system and platform, while the customer still manages its data, including encryption options, classifies its assets and sets permissions with identity and access management. Some controls are shared: AWS patches its infrastructure while customers patch their guest systems and applications, and each side configures and trains its own part.

Microsoft publishes a responsibility matrix. Part of it, for the cloud models:

Microsoft explains the shared rows. In PaaS it provides baseline network security, but you configure application-level network controls. In PaaS and SaaS it manages parts of the application stack, but application configuration, code security and access controls stay with you. The page also notes that it describes who is expected to configure and operate each control, and that it is illustrative guidance, not a legal conclusion: your contract with a provider governs.

Google's framework says that in IaaS the bulk of security responsibility is the customer's, that in SaaS the bulk is Google's, and that customers always remain responsible for their access policies and data. Google adds a term of its own, shared fate, for the idea that provider and customer should work together continuously to improve the customer's security rather than simply divide duties and walk away.

The point: the providers phrase it three ways, but the shape is identical. The provider's share grows from IaaS to SaaS; your data, your identities and your configuration never leave your side of the line.

What never moves

Look again at the first row of Microsoft's matrix. Data, configuration and identities belong to the customer in every model. Microsoft's page adds that you always remain responsible for the endpoints that reach your services and for access management. That means role-based access, multifactor authentication and the rules about who may sign in from where. This explains why so many cloud incidents are customer incidents. A storage bucket left readable by anyone, an administrator account without multifactor authentication, a role that grants far more than its job needs: each is a configuration of something the customer controls, in every service model. The provider supplies the tools, such as identity services, encryption and logging. Using them well is your work, and Lessons 25 to 30 teach it.

The questions that choose a model

When an exam scenario asks for the appropriate service model, run through these questions in order.

1. Does a standard product already do the job? If so, SaaS is usually fastest and least work. Check where it stores data and how it connects to your other systems.
2. Does any requirement reach below the application? A specific operating system, a kernel module, a driver or an agent you must install all point to IaaS.
3. Is it custom code on a supported language with no special host needs? PaaS takes the operating system and scaling work off your hands.
4. Is the work short, triggered by events and stateless? FaaS fits, within its limits on run time and memory.
5. Who will operate it? Every step to the left adds patching, monitoring and on-call duty for someone on your team.
6. How portable must it be? The further right you go, the more your design leans on one provider's platform. Lesson 16 returns to that cost, vendor lock-in.

Common misconceptions

• "Serverless means there are no servers." Your code still runs on servers; the provider owns, patches and scales them. Lambda's documentation describes running code without provisioning or managing servers, not without servers.
• "With SaaS, the provider is responsible for my data." Every provider's model leaves your data, your user accounts and your access configuration with you, even in SaaS.
• "In IaaS, the provider patches my virtual machines." AWS's page assigns the guest operating system, including updates and security patches, to the customer. The provider patches the hypervisor and hardware beneath it.
• "FaaS is just PaaS with a new name, so the same limits apply." It is related to PaaS, but its unit is a short, stateless function with a hard run-time ceiling, which rules it out for long jobs that a PaaS application could handle.

Recap

• NIST SP 800-145 (2011) defines cloud computing by five characteristics: on-demand self-service, broad network access, resource pooling, rapid elasticity and measured service.
• IaaS gives you virtual infrastructure and its operating system; PaaS takes the operating system and runtime; FaaS takes everything but a short, event-triggered function; SaaS takes everything but user-level settings.
• Choose the model by the lowest layer a requirement touches, by who will operate it, and by limits such as FaaS's 15-minute run time on Lambda.
• Shared responsibility: the provider's share grows from IaaS to SaaS, but data, identities, access and configuration always stay with the customer.

Sources

1. Mell, P., and Grance, T. (2011). The NIST definition of cloud computing (NIST Special Publication 800-145). National Institute of Standards and Technology. csrc.nist.gov
2. Amazon Web Services. (n.d.). Shared responsibility model. Retrieved 27 October 2026. aws.amazon.com
3. Microsoft. (2026). Shared responsibility in the cloud. Microsoft Learn. learn.microsoft.com
4. Google Cloud. (2023). Shared responsibilities and shared fate on Google Cloud. Google Cloud Architecture Framework. docs.cloud.google.com
5. Amazon Web Services. (n.d.). What is AWS Lambda? and Lambda quotas. AWS Lambda Developer Guide. docs.aws.amazon.com (overview); docs.aws.amazon.com (quotas)
6. Amazon Web Services. (n.d.). What is AWS Elastic Beanstalk? AWS Elastic Beanstalk Developer Guide. docs.aws.amazon.com
7. Google Cloud. (n.d.). An overview of App Engine and Compare AWS and Azure services to Google Cloud. docs.cloud.google.com (App Engine); docs.cloud.google.com (comparison)
8. CompTIA. (2023). CompTIA Cloud+ CV0-004 certification exam: Exam objectives (Version 2.0), objective 1.1. CompTIA (PDF)

A strategy signed on 8 February 2011

On 8 February 2011 Vivek Kundra, then the US federal Chief Information Officer, published the Federal Cloud Computing Strategy. Its numbers were blunt. Agencies were using less than 30 percent of their available server capacity, largely because every system kept reserve capacity for peaks. An estimated $20 billion of the government's $80 billion in annual IT spending was a potential target for migration to cloud computing. The strategy announced a Cloud First policy: before making any new investment, an agency had to evaluate safe, secure cloud options. It also noted that cloud computing could be implemented through several deployment models, "private, community, public, or a hybrid combination."

Following that one government through the next decade is a quick way to learn the deployment models, because it ended up using all of them. This lesson covers CV0-004 objective 2.1, which compares public, private (including on-premises), hybrid and community deployment, together with two ideas from objective 1.2 that only make sense alongside them: cloud bursting and multicloud tenancy.

Public cloud: open to anyone who pays

NIST's definition, published seven months after the strategy, says a public cloud is provisioned for open use by the general public. It may be owned and operated by a business, academic or government organization, and it exists on the premises of the cloud provider. Public clouds are multi-tenant: in NIST's resource-pooling sense, your virtual machines may share physical hardware with other customers' machines, kept apart by virtualization and the provider's controls.

The strategy described the economics it was after. Organizations could measure and pay for only the IT resources they consumed, and increase or decrease their usage to match requirements and budget, instead of buying peak capacity that sits idle most of the year. What a public cloud asks in return is acceptance: of the provider's terms and price changes, of its outages, of shared infrastructure, and of a location you can choose only down to the level of a region.

Private cloud: DISA builds its own

One of the strategy's own case studies was a private cloud. The Defense Information Systems Agency had built RACE, a private infrastructure-as-a-service platform, having judged that a private solution would deliver the gains in efficiency and agility while keeping strict security controls. The strategy reports that provisioning a dedicated server environment used to take three to six weeks, and that with RACE a user could have working service space in 24 hours. RACE also kept applications, databases and web servers separate from one another and had a strict cleansing process for removing an application from the platform.

NIST defines a private cloud as one provisioned for the exclusive use of a single organization. It may be owned and operated by that organization, by a third party or by both, and it may exist on or off premises. CompTIA's objective lists "on premises" under private, which is the commonest form but not the only one: a provider can run dedicated infrastructure for one customer in its own building, and that is still a private cloud. What makes it a cloud rather than a virtualized server room is the set of characteristics from Lesson 2: self-service provisioning, pooling, elasticity and metering, usually behind an API. Open-source projects such as OpenStack exist to supply exactly that layer; the project describes OpenStack as a cloud operating system that controls large pools of compute, storage and networking resources throughout a data center, all provisioned through APIs.

The cost is that you own the capacity. A private cloud can be only as elastic as the hardware you bought, which brings back the original problem: buy for the peak and leave it idle, or run out at the peak.

Why this matters: "private" describes who may use the infrastructure, not where it stands. On-premises is the commonest private cloud, but a dedicated environment hosted by a provider is private too.

Community cloud: shared by those with shared concerns

The 2011 strategy also pointed to NASA's Nebula, which it said gave researchers IT services through a community cloud in minutes, where procurement used to take months. NIST's definition explains the term: a community cloud is provisioned for the exclusive use of a specific community of consumers from organizations with shared concerns, such as mission, security requirements, policy and compliance considerations. It may be run by community members, by a third party or by both, on or off premises.

Today's clearest examples are the government regions the large providers run. AWS GovCloud (US) consists of isolated AWS Regions designed so that US government agencies and customers can move sensitive workloads into the cloud. Those regions are designed to meet requirements such as FedRAMP High, Department of Defense Impact Levels 4 and 5, and the FBI's CJIS policy. To help customers with US export-control rules such as ITAR, those Regions are administered only by AWS personnel who are US citizens. Azure Government uses physically isolated data centers and networks located only in the US, and it limits potential access to systems processing customer data to screened US persons. Microsoft also validates the eligibility of every customer, who must be a US federal, state or local government body or one of its partners. Neither provider markets these as community clouds, but each matches NIST's definition: one community, bound by shared compliance obligations, sharing infrastructure that the general public cannot use.

Hybrid cloud and the burst

NIST's hybrid cloud is a composition of two or more distinct cloud infrastructures, private, community or public, that remain separate entities but are bound together by technology that makes data and applications portable between them. Its one example is cloud bursting for load balancing between clouds.

Work a burst through. A retailer runs its web tier in a private cloud sized for 400 web server instances. That covers every week of the year except the week of its biggest sale, when it needs 650. Buying 250 more servers for one week a year would repeat exactly the utilization problem the 2011 strategy described. Instead the retailer bursts: when private capacity runs out, automation starts 250 instances in a public cloud and the load balancer sends the overflow there. When demand falls, the public instances are destroyed and their billing stops.

That short story hides five requirements, and scenario questions like to test them.

1. Portable workloads. The same machine image or container image must run in both clouds.
2. Connectivity. A VPN or dedicated connection, the subject of Lesson 6, joins the two networks, and their address ranges must not overlap.
3. Data placement. Burst instances that query a database in the private cloud add a round trip across the link to every request. Read-heavy data may need a replica or cache near the public instances.
4. Consistent identity and security policy. The same people, roles and firewall rules must apply on both sides.
5. Automation in both directions. Scaling out without scaling back in turns a one-week burst into a permanent bill.

Providers also sell hybrid from the other direction: their cloud, installed in your building. AWS Outposts is a fully managed service that places AWS compute and storage capacity at a customer site. AWS operates, monitors and manages it as part of an AWS Region, and it uses the same programming interfaces, for workloads that need low latency or local data processing. Microsoft's Azure Local extends Azure capabilities to customer-owned environments, managed through Azure Arc, for compute that must remain on premises, low-latency decisions or particular compliance requirements.

Why this matters: hybrid is not simply some servers here and some in the cloud. NIST's word is bound: the environments must be joined by technology that lets workloads and data move, and a burst is the test of whether they really are.

Multicloud: the Pentagon changes its mind

On 6 July 2021 the US Department of Defense cancelled its Joint Enterprise Defense Infrastructure cloud solicitation, known as JEDI, and began terminating the contract. The department said that "due to evolving requirements, increased cloud conversancy, and industry advances" JEDI no longer met its needs, and its acting Chief Information Officer, John Sherman, pointed to changed user requirements to leverage multiple cloud environments. In its place the department announced the Joint Warfighting Cloud Capability, JWCC, as a multi-cloud, multi-vendor contract. On 7 December 2022 it awarded JWCC contracts to four providers, Amazon Web Services, Google, Microsoft and Oracle, to supply commercial cloud services at all classification levels, from headquarters to the tactical edge.

Multicloud means using services from more than one public cloud provider at the same time. It is a different axis from hybrid, which joins different kinds of cloud, and an organization can be both at once. The reasons usually given for multicloud are bargaining power against lock-in, the best service for each job, customer or regulatory requirements, and resilience against one provider's failure. The costs are just as concrete: two or more sets of skills and tools, different identity and network models to secure, and the expense and delay of moving data between providers.

Tenancy across clouds

Objective 1.2 lists multicloud tenancy under service availability, so be precise about what a tenant is. Every public cloud is multi-tenant: many customers share pooled hardware while each customer's resources and data stay isolated. Each provider gives you a unit that holds your resources, isolates them and carries their bill.

Multicloud tenancy means holding such units at more than one provider. For availability it helps only if the design uses it. A copy of a workload in a second provider protects you from the first provider's regional outage only if that copy is ready. It needs current data, working identity, an address that DNS can point users to, and a tested way to take the traffic. Many multicloud estates are really several single-cloud systems side by side, each dependent on one provider. That is useful for negotiation and choice, but it is not a disaster recovery plan. Lesson 4 turns the question into numbers with recovery time and recovery point objectives.

Why this matters: a second cloud adds availability only when data, identity, DNS and failover are built for it; otherwise it adds cost and complexity without protection.

Choosing a model

Common misconceptions

• "A private cloud is by definition on premises." NIST defines it by exclusive use and allows it on or off premises, run by the organization or a third party.
• "Any mix of on-premises servers and a cloud account is a hybrid cloud." NIST requires the environments to be bound together by technology that enables data and application portability. Two unconnected estates are just two environments.
• "Multicloud is automatically more available." Only if the second provider holds current data, identity, DNS and a tested failover path.
• "A virtualized data center is a private cloud." Without self-service, pooling, elasticity and metering it is virtualization, not a cloud.

What you now know

• Public clouds are open to anyone and multi-tenant; private clouds serve one organization, on or off premises; community clouds serve a group with shared concerns, as government regions such as AWS GovCloud (US) and Azure Government do.
• Hybrid clouds bind two or more distinct clouds for portability; cloud bursting sends peak load from private to public capacity and needs portable images, connectivity, good data placement, consistent security and automation both ways.
• Multicloud uses several public providers; the US Department of Defense moved from the single-vendor JEDI plan to the four-vendor JWCC contract in 2022.
• Tenancy units differ by provider (account, subscription, project), and a second tenancy improves availability only when it is built and tested for failover.

Sources

1. Kundra, V. (2011). Federal cloud computing strategy. The White House, Office of Management and Budget. obamawhitehouse.archives.gov (PDF)
2. Mell, P., and Grance, T. (2011). The NIST definition of cloud computing (NIST Special Publication 800-145). National Institute of Standards and Technology. csrc.nist.gov
3. OpenInfra Foundation. (n.d.). OpenStack software. openstack.org
4. Amazon Web Services. (n.d.). What is AWS GovCloud (US)? AWS GovCloud (US) User Guide. docs.aws.amazon.com
5. Microsoft. (2022). Azure Government overview. Microsoft Learn. learn.microsoft.com
6. Amazon Web Services. (n.d.). What is AWS Outposts? AWS Outposts User Guide. docs.aws.amazon.com
7. Microsoft. (n.d.). About Azure Local. Microsoft Learn. learn.microsoft.com
8. US Department of Defense. (2021, July 6). Future of the Joint Enterprise Defense Infrastructure cloud contract [Press release]. war.gov
9. US Department of Defense. (2022, December 7). Department of Defense announces Joint Warfighting Cloud Capability procurement [Press release]. war.gov
10. Amazon Web Services, Google Cloud and Microsoft. (n.d.). What is AWS Organizations?; Resource hierarchy; Organize your Azure resources effectively. docs.aws.amazon.com; docs.cloud.google.com; learn.microsoft.com

An empty DNS record at 11:48 PM

At 11:48 PM Pacific time on 19 October 2025, customers of Amazon DynamoDB in AWS's Northern Virginia region, us-east-1, began seeing errors. AWS's own post-event summary explains why. DynamoDB keeps hundreds of thousands of DNS records for its load balancers, maintained by an automated system split into a planner and an enactor, and the enactor ran redundantly and independently in three availability zones. A latent race condition in that system left the regional endpoint, dynamodb.us-east-1.amazonaws.com, with an empty DNS record that the automation failed to repair. Clients could not find the service, so customers and other AWS services that depended on DynamoDB could not open new connections to it. The damage spread through dependencies: between 2:25 AM and 10:36 AM new EC2 instance launches failed, Network Load Balancer connection errors ran from 5:30 AM until 2:09 PM, and the event did not end until 2:20 PM on 20 October.

Hold that event in mind and ask the questions an architect has to answer. The system that failed was spread across three availability zones, so why did zones not save it? Which customer designs kept working? And how much downtime, and how much lost data, should a business plan to tolerate before it pays for something better? This lesson works through those questions. It covers the rest of CV0-004 objective 1.2: regions, availability zones, edge computing, availability monitoring, and disaster recovery with RTO, RPO and hot, warm and cold sites. Cloud bursting and multicloud tenancy, the other parts of 1.2, were in Lesson 3.

Regions and zones: the failure domains you can choose

A failure domain is a set of resources that can fail together. Providers organize their infrastructure into nested failure domains, and much of availability design is choosing which ones your workload spans.

The numbers behind the words matter. Microsoft says its zones are typically several kilometers apart and usually within 100 kilometers of each other, close enough for a round-trip latency target of less than about 2 milliseconds, which is fast enough for synchronous replication between zones. AWS connects its zones over low-latency, high-bandwidth, redundant metro fiber and notes that, because zones are physically separate, a fire, tornado or flood would affect only one. Regions, by contrast, are far apart, which protects against regional disasters but makes synchronous replication between them impractical.

Microsoft draws one more useful distinction. A zone-redundant resource is spread across zones by the service itself, which handles failover. A zonal resource is pinned to one zone you choose; it is isolated from faults in other zones, but if its zone fails, you must have built the failover yourself.

Working the problem: what zones protect, and what they do not

Now return to October 2025. Zones protect against failures of things that exist separately in each zone: a power feed, a cooling plant, a network link, a building. The DynamoDB fault was in something shared by the whole region: one regional DNS plan and one regional endpoint name. Running the enactor in three zones gave it three copies of the same logic, and the race condition was in the logic. Every zone was exposed at once. Microsoft's guidance states the general rule plainly: availability zones don't protect against a full-region outage.

So the customers who stayed up through events like this are the ones whose critical paths did not depend on that one region. That means a second region with its own copy of the application, data replicated to it, and a way to move users there. Multi-region design is expensive and complicated, which is why Microsoft reserves the recommendation for mission-critical workloads. For everything else the honest answer is to choose a target: decide how much downtime and data loss the business can accept, and design to that.

What matters here: zones protect against a facility failing; regions protect against a region failing, including a fault in shared regional software. Spreading across zones is the default for production, and a second region is the answer only when the business case demands it.

Putting numbers on availability

Availability is usually stated as a percentage of time a service works. Each extra nine cuts the allowed downtime by a factor of ten. Using a 365-day year and a 730-hour average month:

Providers publish these targets as service level agreements, and the design changes the promise. AWS's compute SLA offers a Region-Level commitment of 99.99 percent monthly uptime for EC2 when all running instances are deployed across two or more availability zones, and an Instance-Level commitment of 99.5 percent for any single instance. The remedy if AWS misses either is a service credit, a percentage of that month's bill, not compensation for your lost business.

Two rules let you combine numbers. When a request needs several components in series, multiply their availabilities: a web tier at 99.99 percent that needs a database at 99.9 percent can be at best 0.9999 times 0.999, about 99.89 percent, which is worse than either part. When components are redundant in parallel, the service fails only if all copies fail, so two independent copies at 99.9 percent give 1 minus 0.001 squared, or 99.9999 percent. The catch is the word independent. October 2025 is a textbook correlated failure: redundant copies that shared one fault failed together, which is why the parallel formula is a ceiling, never a promise.

Watching availability from the right place

Availability monitoring means checking continuously whether a service actually works for its users. It has three layers. Health checks, run by load balancers and orchestrators, test each instance and remove failed ones from service. Synthetic checks run scripted requests against the public endpoint, the way a user would, and record success and response time. And provider status pages, such as the AWS Health Dashboard, Azure status and Google Cloud Service Health, report incidents in the provider's own services.

The design rule is that monitoring must not share the failure domain it watches. A checker running in the same region as the application goes silent in exactly the outage you most need to hear about, so at least one synthetic check should run from somewhere else, and alerts should reach people by a path that does not depend on the failed region.

The edge: moving compute toward users

Edge computing runs processing close to the people or devices that use it, instead of in a distant region. It answers two problems a region cannot: latency, because light in fiber takes time to cross a continent, and bandwidth, because shipping every raw sensor reading or video frame to a region is expensive. AWS's catalogue shows the spectrum. Local Zones place compute and storage closer to end users in large population and industry centers. Wavelength Zones deploy compute and storage at the edge of telecommunication carriers' 5G networks for ultra-low latency to 5G devices. Content delivery networks, in Lesson 7, push cached content and small functions to edge locations, and IoT gateways, in Lesson 15, process sensor data on site. The cost is operational: many small sites to deploy to, patch and monitor instead of a few large ones.

Recovery objectives: how long, and how much

Disaster recovery planning starts from two numbers, defined in NIST's contingency planning guide, SP 800-34 Revision 1.

• Recovery time objective (RTO): the maximum time a system can remain unavailable before the impact on other systems and on the business becomes unacceptable. It answers: how long may recovery take?
• Recovery point objective (RPO): the point in time, before the disruption, to which data must be recovered. It answers: how much recent data may be lost?

Above both sits the maximum tolerable downtime, the total outage leaders are willing to accept for the business process. NIST describes the RTO as chosen so that recovery fits within it.

Work an example. An order database is backed up every night at 01:00 to another region, and a full restore takes three hours. On a Tuesday the database is lost at 17:00. Restoring the 01:00 backup brings back the data as it stood sixteen hours earlier, so sixteen hours of orders are gone; had the failure struck at 00:59, nearly twenty-four hours would be. The restore itself takes three hours, plus however long it takes to notice the failure and decide to recover. If the business set an RTO of four hours and an RPO of one hour, this design meets the first and fails the second badly.

Now change one input. Add continuous replication from the primary database to a standby in the second region. The data loss shrinks to whatever the replication lag was at the moment of failure, typically seconds to minutes, and the RPO is met. The RTO now depends on what else is waiting in the second region, which is the question the site types answer.

Hot, warm and cold sites, and their cloud versions

NIST describes alternate processing sites by how ready they are.

In the cloud you do not lease floor space, but the same ladder exists, and AWS's disaster recovery whitepaper names four rungs, ordered from cheapest and slowest to costliest and fastest. Backup and restore keeps backups in the recovery region and builds everything after the disaster, much like a cold site. Pilot light keeps data replicated to live databases and core infrastructure ready but switched off, so servers must be turned on and scaled before taking traffic. Warm standby keeps a scaled-down but fully working copy running; as the whitepaper puts the difference, pilot light cannot process requests without additional action, whereas warm standby can handle traffic at reduced capacity immediately. Multi-site active/active serves users from more than one region at all times, the cloud form of NIST's hot site, which the guide illustrates with two identical systems in production at alternate locations, each built for the full load.

What matters here: RPO is set by how often data leaves the primary site, and RTO by how much is already running at the recovery site. Pay for the rung the business's numbers require, and test the recovery, because an untested plan has an unknown RTO.

Common misconceptions

• "Three availability zones make a workload immune to outages." Zones protect against facility failures. A fault in shared regional software or a regional endpoint, as in October 2025, reaches every zone at once.
• "RPO is how long recovery takes." That is RTO. RPO is about data: how far back in time the recovered data may be.
• "An SLA guarantees the uptime it states." It defines a target and a remedy. AWS's EC2 SLA pays service credits against the bill; your losses are yours to design against.
• "Two redundant copies always multiply away the risk." Only if their failures are independent. Copies that share software, configuration or a regional dependency can fail together.

The short version

• Regions are isolated geographic areas; availability zones are physically separate data center groups inside a region with independent power, cooling and networking; edge locations move compute close to users and devices.
• Multi-zone is the production default and survives a facility failure; multi-region survives a regional failure but costs more and needs replicated data and a tested traffic switch.
• Know the nines (99.9 percent is about 43.8 minutes a month, 99.99 percent about 4.4), multiply availabilities in series, and treat parallel redundancy as a ceiling.
• Monitor availability from outside the failure domain, with health checks, synthetic checks and provider status pages.
• RTO is how long recovery may take, RPO how much data may be lost, and both must fit the maximum tolerable downtime. Cold, warm and hot sites, or backup and restore, pilot light, warm standby and active/active, trade cost for speed.

Sources

1. Amazon Web Services. (2025). Summary of the Amazon DynamoDB service disruption in the Northern Virginia (US-EAST-1) Region. aws.amazon.com
2. Amazon Web Services. (n.d.). AWS Regions and Availability Zones. AWS Global Infrastructure documentation. docs.aws.amazon.com
3. Microsoft. (2026). What are Azure availability zones? Microsoft Learn. learn.microsoft.com
4. Google Cloud. (n.d.). Geography and regions. docs.cloud.google.com
5. Amazon Web Services. (n.d.). Amazon Compute Service Level Agreement. Retrieved 27 October 2026. aws.amazon.com
6. Swanson, M., Bowen, P., Phillips, A. W., Gallup, D., and Lynes, D. (2010). Contingency planning guide for federal information systems (NIST Special Publication 800-34 Revision 1). National Institute of Standards and Technology. csrc.nist.gov
7. Amazon Web Services. (n.d.). Disaster recovery options in the cloud. In Disaster recovery of workloads on AWS: Recovery in the cloud [Whitepaper]. docs.aws.amazon.com
8. The Kubernetes Authors. (n.d.). Taints and tolerations and Well-known labels, annotations and taints. Kubernetes documentation. kubernetes.io (taints); kubernetes.io (labels)
9. Amazon Web Services, Microsoft and Google. (n.d.). AWS Health Dashboard; Azure status; Google Cloud Service Health. health.aws.amazon.com; azure.status.microsoft; status.cloud.google.com

A /16 and a pencil

Every virtual private cloud begins as one block of private addresses written in CIDR notation. AWS lets that block, and every subnet in it, be anything from a /16, which holds 65,536 IPv4 addresses, down to a /28, which holds sixteen. Azure's smallest subnet is a /29. Choose the block carelessly and you will discover the mistake months later, when you try to connect this network to another one and the provider refuses because the ranges overlap.

A VPC, called a virtual network or VNet in Azure, is a private, isolated network that a provider builds for you in software. Inside it you create subnets, attach route tables that decide where traffic goes, and add gateways to the internet and to other networks. This lesson builds one end to end, then changes a single input to show how a design fails. It covers the VPC half of CV0-004 objective 1.3: the virtual private cloud, peering and transit gateways, subnets, and routing and switching, including VLANs, software-defined networking, static routes and route tables. Lesson 6 covers connections into the VPC, and Lesson 7 the services that stand in front of it. If subnet masks and binary still feel slow, the subnetting lessons of Network+ Certification Prep (CS 215) are the place to practise.

What a VPC is, underneath

In your own data center you would separate networks with VLANs: the IEEE 802.1Q tag on each frame carries a 12-bit VLAN identifier, and switches keep traffic with different tags apart. A cloud provider cannot simply give every customer VLANs. RFC 7348, which defines VXLAN, explains why: with a 12-bit identifier the limit is 4094 VLANs, and for a multi-tenant data center serving many customers that limit is often inadequate. VXLAN instead wraps each customer's traffic in an overlay identified by a 24-bit network identifier, which allows about sixteen million separate segments over the same physical network.

The other half of the trick is software-defined networking. RFC 7426 describes SDN as a programmable-networks approach that separates the control plane, which decides where traffic should go, from the forwarding plane, which moves the packets. In a VPC you never touch a switch or a router. You change the control plane through the provider's API, by creating subnets and editing route tables, and the provider's hosts forward packets accordingly. That is also why some on-premises habits do not carry over. AWS, for example, does not support broadcast in a VPC at all, and reserves each subnet's broadcast address for that reason. In the exam's phrase "routing and switching", switching is something the provider does for you; routing is something you configure.

Step 1: choose the address block

Start with the private ranges from RFC 1918: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Then check what the new network must one day talk to. Suppose the company's offices use 10.0.0.0/16 and an existing analytics VPC uses 10.30.0.0/16. Pick 10.20.0.0/16 for the new production VPC: large enough to grow, and overlapping nothing. Write that choice into a central address plan, which is what IP address management means in practice, so the next team does not reuse it.

Step 2: carve subnets by zone and by tier

Use three tiers, public for load balancers, application for servers and data for databases, and repeat each tier in two availability zones so one zone can fail without taking a tier with it.

Why 251 and not 254? AWS reserves five addresses in every subnet. In 10.20.0.0/24 they are 10.20.0.0, the network address; 10.20.0.1, the VPC router; 10.20.0.2, used for the provider's DNS; 10.20.0.3, held for future use; and 10.20.0.255, the broadcast address, reserved even though broadcast is unsupported. The providers differ in the details, and the differences are exam material.

Remember: in AWS, "spread across zones" means one subnet per zone, because a subnet lives in exactly one Availability Zone. In Azure and Google Cloud a subnet is regional, so zone spreading happens at the level of the resources inside it.

Step 3: route tables decide public and private

A route table is a list of destinations and targets, and every subnet is associated with one. Every route table starts with a local route covering the whole VPC range, which AWS describes as the default route for communication within the VPC. What else you add decides the subnet's character. AWS's definition is exact: a public subnet has a direct route to an internet gateway; a private subnet does not.

The NAT gateway is what lets the application tier fetch updates safely. AWS describes it as letting instances in a private subnet connect to services outside the VPC while those instances cannot receive unsolicited inbound connections. In production you would place one NAT gateway in each zone's public subnet, so losing zone a does not cut zone b off from its updates.

When more than one route matches a packet, the most specific one wins. RFC 4632, the CIDR standard, states it simply: forwarding is done on a longest-match basis. A packet for 172.16.8.20, facing a route to 172.16.0.0/12 through a VPN and a route to 172.16.8.0/24 through a peering connection, takes the peering connection, because a /24 is more specific than a /12, and both beat the 0.0.0.0/0 default. The routes you type into a table are static routes; Lesson 6 adds dynamic routes learned through BGP.

Step 4: connect a second VPC with peering

The analytics team's VPC, 10.30.0.0/16, needs to query the production databases. VPC peering joins two VPCs so they exchange traffic privately over the provider's network. Peering alone moves nothing: you also add a route in each side's tables, 10.30.0.0/16 toward the peering connection in production and 10.20.0.0/16 toward it in analytics, and you open the firewall rules for exactly the traffic needed.

All three providers attach the same two conditions to peering. The address ranges may not overlap: AWS will not create a peering connection if any of the VPCs' CIDR blocks overlap, and Google checks peered networks for overlapping subnet ranges. And peering is not transitive: AWS says it does not support transitive peering relationships, Google says VPC Network Peering does not provide transitive routing, and Microsoft calls peering connections nontransitive.

Step 5: when peering stops scaling, use a hub

Non-transitive peering means that every pair of networks that must talk needs its own peering connection. A full mesh of n networks needs n(n minus 1) divided by 2 connections: 6 networks need 15, 15 networks need 105, and each one has routes and rules to maintain. The answer is a hub. AWS Transit Gateway is a network transit hub that interconnects VPCs and on-premises networks: each VPC or VPN connects to it as an attachment, and the hub's own route tables decide which attachments can reach which. Azure's reference designs use a hub-and-spoke topology in which spokes peer with a hub virtual network that hosts shared components such as a firewall appliance or a VPN gateway, and spoke-to-spoke traffic flows through the hub.

The second run: one input changes, and the design breaks

Now replay step 4 with one difference. The analytics team, working from a tutorial, built its VPC as 10.20.0.0/16, the same range as production. The peering request fails, and it should: if both sides own 10.20.11.5, no route can say which one a packet means. There are workarounds involving address translation, but the durable fix is to re-address one of the VPCs from the central plan, which is far cheaper before workloads move in than after.

Change a different input. Production peers with analytics, analytics peers with a reporting VPC, and someone expects production to reach reporting through analytics. It cannot, because peering is not transitive, and adding routes that point through the middle VPC will not change that. The fixes are a direct peering between production and reporting, or attaching all three to a transit hub.

Remember: plan addresses before anything else, keep every range unique across everything you might ever connect, and draw the connection graph: if any path passes through a middle VPC, peering alone will not carry it.

Common misconceptions

• "A subnet is public because its instances have public IP addresses." A public address is useless without a route to an internet gateway. The route table decides.
• "If A peers with B and B peers with C, then A can reach C." Peering is non-transitive at all three providers.
• "A NAT gateway lets the internet reach private servers." It carries connections that start inside and go out. Inbound connections from the internet are refused.
• "A cloud subnet behaves like an office VLAN." There is no switch to configure, and AWS does not support broadcast in a VPC; the provider's SDN does the switching.

Putting it together

• A VPC is a software-defined, isolated network; providers use overlays such as VXLAN, with its 24-bit identifier, because 4094 VLANs cannot serve many tenants.
• Choose a non-overlapping RFC 1918 block, then subnets per tier and per zone; AWS and Azure reserve five addresses per subnet, Google four.
• AWS subnets live in one zone; Azure virtual networks are regional and span zones; Google VPC networks are global with regional subnets.
• Route tables make subnets public (default route to an internet gateway), private (default route to a NAT gateway) or isolated (local route only), and the longest matching prefix wins.
• Peering needs non-overlapping ranges and is non-transitive; a transit gateway or hub-and-spoke design replaces a growing mesh.

Sources

1. Amazon Web Services. (n.d.). Subnet CIDR blocks, Subnets for your VPC, Route tables and NAT gateways. Amazon VPC User Guide. docs.aws.amazon.com (sizing); docs.aws.amazon.com (subnets); docs.aws.amazon.com (route tables); docs.aws.amazon.com (NAT)
2. Amazon Web Services. (n.d.). VPC peering basics and What is AWS Transit Gateway? docs.aws.amazon.com (peering); docs.aws.amazon.com (transit gateway)
3. Microsoft. (n.d.). Azure Virtual Network frequently asked questions and Hub-spoke network topology in Azure. Microsoft Learn. learn.microsoft.com (FAQ); learn.microsoft.com (hub-spoke)
4. Google Cloud. (n.d.). VPC networks, Subnets and VPC Network Peering. docs.cloud.google.com (VPC); docs.cloud.google.com (subnets); docs.cloud.google.com (peering)
5. Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and Wright, C. (2014). Virtual eXtensible Local Area Network (VXLAN): A framework for overlaying virtualized layer 2 networks over layer 3 networks (RFC 7348). RFC Editor. rfc-editor.org
6. Haleplidis, E., Pentikousis, K., Denazis, S., Hadi Salim, J., Meyer, D., and Koufopavlou, O. (2015). Software-defined networking (SDN): Layers and architecture terminology (RFC 7426). RFC Editor. rfc-editor.org
7. Fuller, V., and Li, T. (2006). Classless inter-domain routing (CIDR): The Internet address assignment and aggregation plan (RFC 4632). RFC Editor. rfc-editor.org
8. Rekhter, Y., Moskowitz, B., Karrenberg, D., de Groot, G. J., and Lear, E. (1996). Address allocation for private internets (RFC 1918). RFC Editor. rfc-editor.org
9. Docker. (n.d.). Bridge network driver. Docker Docs. docs.docker.com

4 October 2021: a network that stopped announcing itself

On 4 October 2021, during routine maintenance on Meta's global backbone, an engineer issued a command meant to assess how much backbone capacity was available. According to the account Meta's Santosh Janardhan published the next day, the command unintentionally took down every connection in the backbone, cutting Facebook's data centers off from one another and from the internet. A tool designed to audit commands like that one had a bug that kept it from stopping it.

What turned a backbone failure into a global disappearance was a design decision in Meta's DNS servers. Their addresses were advertised to the rest of the internet with the Border Gateway Protocol, and, to keep users away from an unhealthy site, each DNS location withdrew its BGP advertisements whenever it could not reach the data centers. With the whole backbone down, every location did exactly that. The DNS servers kept running, but the internet no longer had a route to them, so nobody could look up Facebook's addresses. Meta's engineers then found that they could not reach the data centers through their normal means, because those networks were down too. The loss of DNS had also broken many of the internal tools they would have used to investigate.

Follow that story one level down and you have most of this lesson. The protocol that made Meta vanish also carries the routes over every dedicated connection between a company and a cloud provider, and the same choice between automatic and manual routing decides how a company's connection fails over. This lesson covers the connectivity half of CV0-004 objective 1.3: public and private connections to the cloud, meaning VPNs and dedicated connections, and BGP.

What BGP actually does

RFC 4271 defines BGP-4 as an inter-Autonomous System routing protocol whose primary function is to exchange network reachability information with other BGP systems. An autonomous system, or AS, is a network under one administration with its own routing policy, identified by an AS number. Two BGP speakers open a session over TCP port 179, and each announces the prefixes it can reach, tagged with the list of autonomous systems the announcement has passed through. When a prefix stops being reachable, its owner sends a withdrawal, and every router that learned the route removes it. Meta's DNS locations did nothing more exotic than send withdrawals.

Cloud connections speak BGP on a small scale. Your edge router and the provider's router form a session; you announce your office and data center prefixes, and the provider announces your VPC ranges. AWS says a device on a Direct Connect link must support BGP and BGP MD5 authentication. Microsoft uses BGP to exchange routes between your on-premises network, your Azure instances and Microsoft's public addresses over ExpressRoute. Google's Cloud Router provides dynamic BGP routing for Cloud VPN, Dedicated Interconnect and Partner Interconnect. Each side of a session needs an AS number, and private networks normally use private AS numbers; AWS's VPN service, for example, lets you choose a private ASN for the Amazon side of the session.

In short: BGP is how two networks tell each other what they can reach. Its speed is what makes failover automatic, and the same speed is what spread Meta's mistake around the world in minutes.

The everyday connection: IPsec across the internet

The quickest way to join an office to a VPC is a site-to-site VPN: encrypted tunnels between a VPN device in the office and the provider's VPN gateway, carried over the internet connection the office already has. The tunnels use IPsec, which RFC 4301 describes as providing integrity, data origin authentication, rejection of replayed packets and confidentiality through encryption for IP traffic.

The providers' versions share a shape. AWS Site-to-Site VPN connects your customer gateway device to a virtual private gateway, which attaches to one VPC, or to a transit gateway. Each VPN connection includes two tunnels so that one can carry traffic if the other is unavailable. Routing can be static or dynamic with BGP, and AWS lists up to 1.25 Gbps for a standard tunnel, with a larger tunnel type reaching 5 Gbps. Azure VPN Gateway sends encrypted traffic between a virtual network and on-premises locations over the public internet. A site-to-site connection is an IPsec/IKE tunnel to an on-premises VPN device. A point-to-site connection, over OpenVPN, IKEv2 or SSTP, lets an individual device connect from a remote location such as home. Google's HA VPN gateway has two interfaces, each with its own external address, and offers a 99.99 percent availability SLA for most topologies.

A VPN's strengths are speed of setup, low cost and encryption included. Its limit is that it rides the internet, so its latency and loss vary with whatever the internet is doing, and its bandwidth is capped per tunnel.

The private connection: a dedicated link

When traffic is heavy or must be steady, companies buy a dedicated connection: a physical circuit into the provider's network that does not cross the public internet. AWS Direct Connect links your internal network to a Direct Connect location over a standard Ethernet fiber-optic cable, bypassing internet service providers. Ports run at 1, 10, 100 or 400 Gbps. Over each you create virtual interfaces: a private one to reach a VPC, or a public one to reach public services such as Amazon S3. Azure ExpressRoute extends your network into Microsoft's cloud over a private connection supplied by a connectivity provider; Microsoft says that because ExpressRoute connections do not go over the public internet, they offer more reliability, faster speeds, consistent latencies and higher security. Each ExpressRoute circuit consists of two connections to two Microsoft Enterprise Edge routers. Google Cloud Interconnect comes in two forms: Dedicated Interconnect, where your network physically meets Google's in a colocation facility over 10, 100 or 400 Gbps circuits, up to eight per connection, and Partner Interconnect, bought through a service provider.

One word in that paragraph misleads people: private. Google states plainly that Cloud Interconnect does not encrypt traffic by default, and offers IPsec tunnels over the interconnect or MACsec on the link. AWS offers MACsec, the IEEE standard for Layer 2 encryption, on Direct Connect, encrypting traffic point to point between your edge device and AWS's. If a regulation requires encryption in transit, a dedicated link alone does not meet it.

Two paths, one protocol: building failover

A common production design combines both. The dedicated connection is the primary path and a site-to-site VPN is the backup, and both run BGP and announce the same prefixes. While both are up, the routers are configured to prefer the dedicated path, for example by announcing more specific prefixes over it, since the longest match wins, as Lesson 5 showed. When the dedicated link fails, its BGP session drops, every route learned over it is withdrawn, and the VPN's routes, already in place, carry the traffic. When the link returns, its routes are announced again and traffic moves back. Nobody edits a route table at 3 AM.

Meta's outage adds three rules for running such a design. First, guard the changes: Meta's audit tool existed precisely to stop a dangerous command and failed because of a bug, so change control and automated checks need testing too. Second, keep a way in that does not depend on the network you are fixing: Meta's engineers could not reach their data centers through normal means, which is the argument for out-of-band management access. Third, rehearse: Meta credits its storm drills, which simulate taking a service, data center or region offline, with its ability to bring systems back carefully. A backup path that has never carried production traffic is a hope, not a plan.

In short: run BGP on both paths so failover is automatic, and prefer the dedicated link while it is healthy. Encrypt the dedicated link if rules demand it, and keep a tested way to manage the network when the network itself is down.

Common misconceptions

• "A dedicated connection is encrypted because it is private." Private means it avoids the public internet. Google says Cloud Interconnect does not encrypt by default, and AWS offers MACsec as an option.
• "A VPN gives guaranteed performance." A site-to-site VPN rides the internet, so its latency and loss vary, and each tunnel has a bandwidth cap.
• "BGP is only for internet service providers." Every dedicated cloud connection uses it, and VPNs use it when they need to fail over or learn new prefixes automatically.
• "One dedicated link is enough for production." A single circuit is a single point of failure, which is why ExpressRoute circuits have two connections, AWS VPN connections have two tunnels, and designs add a second link or a VPN backup.

Where this leaves us

• BGP exchanges reachability between autonomous systems over TCP port 179; announcements add routes and withdrawals remove them, as Meta's DNS servers showed on 4 October 2021.
• Site-to-site VPNs build IPsec tunnels over the internet: quick, cheap and encrypted, but variable and capped per tunnel. Point-to-site VPNs connect individual devices.
• Dedicated connections (Direct Connect, ExpressRoute, Cloud Interconnect) give consistent latency and high bandwidth, use BGP, and are not encrypted unless you add MACsec or IPsec.
• Pair a dedicated primary with a VPN backup, run BGP on both, prefer the dedicated path, and rehearse the failover with out-of-band access ready.

Sources

1. Janardhan, S. (2021, October 5). More details about the October 4 outage. Engineering at Meta. engineering.fb.com
2. Rekhter, Y., Li, T., and Hares, S. (Eds.). (2006). A Border Gateway Protocol 4 (BGP-4) (RFC 4271). RFC Editor. rfc-editor.org
3. Kent, S., and Seo, K. (2005). Security architecture for the Internet Protocol (RFC 4301). RFC Editor. rfc-editor.org
4. Amazon Web Services. (n.d.). What is AWS Site-to-Site VPN? and Site-to-Site VPN quotas. docs.aws.amazon.com (VPN); docs.aws.amazon.com (quotas)
5. Amazon Web Services. (n.d.). What is Direct Connect? and MAC Security (MACsec). Direct Connect User Guide. docs.aws.amazon.com (Direct Connect); docs.aws.amazon.com (MACsec)
6. Microsoft. (n.d.). Azure ExpressRoute overview and What is Azure VPN Gateway? Microsoft Learn. learn.microsoft.com (ExpressRoute); learn.microsoft.com (VPN Gateway)
7. Google Cloud. (n.d.). Cloud Interconnect overview, Dedicated Interconnect overview and Cloud VPN overview. docs.cloud.google.com (Interconnect); docs.cloud.google.com (Dedicated); docs.cloud.google.com (VPN)
8. RIPE NCC. (n.d.). RIPEstat Data API. stat.ripe.net

Two balancers, two layers

AWS's documentation describes its Network Load Balancer as functioning at the fourth layer of the OSI model and handling millions of requests per second. Its Application Load Balancer functions at the seventh layer, the application layer, where it evaluates listener rules against each HTTP request, its path, its host header, before choosing a target. Those two sentences hold the most important comparison in this lesson. A device in front of your servers can only make decisions about what it reads, and reading more costs more work per request.

This lesson covers the network functions in CV0-004 objective 1.3: application load balancers, network load balancers, application gateways, content delivery networks and firewalls. Network+ Certification Prep (CS 215) introduces load balancers, proxies and CDNs on premises; here you meet their cloud forms and the differences between providers. The table is the lesson in one view, and the sections after it read its rows.

Reading the load balancer rows

A layer 4 balancer sees an incoming flow's addresses and ports and nothing inside it. It cannot route /api one way and /images another, because it never reads the path. But it is fast, it carries any TCP or UDP protocol, and depending on how targets are registered it can preserve the client's source address. Microsoft describes Azure Load Balancer the same way: it operates at layer 4, is the single point of contact for clients, and distributes inbound flows to backend instances according to rules and health probes.

A layer 7 balancer terminates the client's connection, reads the request and opens its own connection to a target. That buys routing by path, host, header or cookie, and it costs one thing people forget: the server now sees the balancer's address as the source. AWS's Application Load Balancer adds an X-Forwarded-For header carrying the client's address, and AWS warns that the header should be used with caution, because a client can send its own forged value.

Three behaviours appear in scenario questions.

• Health checks. Both kinds probe each target and stop sending traffic to one that fails, then resume when it recovers. This is what lets one server die without users noticing.
• Cross-zone load balancing. A balancer has a node in each zone. With cross-zone balancing, every node spreads traffic across all targets in all zones; without it, each node uses only its own zone's targets. AWS's example has 2 targets in one zone and 8 in another: with cross-zone balancing each of the 10 receives 10 percent, and without it the zone with 2 targets gives each 25 percent while the other zone's targets get 6.25 percent each. AWS keeps cross-zone balancing always on at the Application Load Balancer level and off by default on Network Load Balancers.
• Routing algorithm. AWS's diagrams use round robin, handing requests to targets in turn, as the default; other algorithms, such as least outstanding requests, favour less busy targets.

The upshot: choose layer 4 for raw speed, non-HTTP protocols and preserved client addresses; choose layer 7 when routing depends on what the request says, and remember to read the client address from a header.

Application gateways: the front door of one application

Microsoft gives the exam's term to a product. Azure Application Gateway is a web traffic load balancer that routes on HTTP attributes, such as URL paths and host headers, where traditional balancers route on address and port. Microsoft's example sends /images requests to servers tuned for images and /video requests to servers tuned for video. Its WAF_v2 tier adds a web application firewall, which inspects requests for attacks such as SQL injection before they reach the application. The older v1 tiers retired on 28 April 2026, a reminder that cloud products have life cycles too, a topic Lesson 33 returns to.

In general terms, an application gateway is a reverse proxy at the edge of one application: it terminates TLS, routes requests by URL and host, applies security policy, and hides the servers behind it. AWS builds the same thing from an Application Load Balancer with AWS WAF attached, and Google from an Application Load Balancer with Cloud Armor. When a question describes regional, path-based routing plus a WAF for a web application, it is describing an application gateway.

CDNs: answering from the edge

AWS's description of CloudFront shows how any CDN works. DNS sends a user's request to the point of presence, or edge location, that can best serve it. If the object is in that location's cache, the edge returns it immediately. If not, the edge requests it from your origin server, starts forwarding the first bytes to the user as they arrive, and keeps a copy for the next request. Azure Front Door describes itself as a content delivery network that accelerates both static and dynamic content globally, and Google's equivalent is Cloud CDN.

What a CDN may keep, and for how long, is governed by HTTP caching rules, RFC 9111, and the origin's headers set them. A real response from Wikipedia's own CDN, observed on 28 October 2026, carried this header: cache-control: s-maxage=86400, must-revalidate, max-age=3600. Under RFC 9111, s-maxage applies only to shared caches such as a CDN, so the edge may treat the page as fresh for 86,400 seconds, a day, while max-age tells a browser to keep its private copy for an hour. The same response carried age: 57543, the cache's estimate of how long ago the copy was fetched or validated: about sixteen hours.

That arithmetic explains the commonest CDN complaint. A retailer changes its price list at the origin, but users keep seeing the old one, because the CDN's copy is still fresh. The fixes are to invalidate the object at the CDN, to give content that changes often a shorter lifetime, or to publish changed files under new, versioned names so the old cache entry is simply never asked for again. Caching helps only cacheable content; a request for a personalized page still travels to the origin, though often over the provider's own network.

Firewalls in a cloud network

Cloud networks offer firewalls at three levels, and the most important difference between them is state.

• Stateful filters at the instance. AWS says plainly that security groups are stateful: if a request is allowed in, the response is allowed out whatever the outbound rules say. Azure network security groups create a flow record for each connection, which Microsoft says is what makes them stateful, and they process rules in priority order from 100 to 4096, lower numbers first, stopping at the first match. Google Cloud tracks connections in its VPC firewall rules, allows return traffic that matches an allowed connection, and describes every VPC network as functioning as a distributed firewall.
• Stateless filters at the subnet. AWS network ACLs are stateless and evaluate rules in order from the lowest number. Because they keep no state, a network ACL that allows inbound HTTPS must also allow the outbound replies, which go to the client's temporary, or ephemeral, port. Forgetting that rule produces connections that open and then hang.
• Managed network firewalls. For central inspection of traffic between networks and to the internet, providers sell firewall services. AWS Network Firewall is a stateful, managed network firewall and intrusion detection and prevention service built on the open-source Suricata engine. Azure Firewall is a fully stateful firewall as a service that inspects both east-west traffic, between internal networks, and north-south traffic, in and out. Google's Cloud NGFW provides the equivalent capabilities on top of its distributed firewall.

Lesson 29 designs rules for these controls in detail, together with web application firewalls and DDoS protection. For architecture, the rule of thumb is layering: stateful rules on every instance, optional stateless filters at subnet edges, and a managed firewall where traffic between zones of trust needs inspection.

The upshot: stateful filters remember connections and allow replies; stateless filters judge every packet alone and need rules for both directions.

One request, front to back

Put the rows in order along a single request. A user's browser resolves the site's name, and DNS points it to the nearest CDN edge. A cached image is answered at the edge; a request for the account page passes through to the origin. At the region it meets an application gateway or layer 7 balancer, whose web application firewall inspects it and whose rules route /account to the right target group, adding X-Forwarded-For on the way. The target's security group admits HTTPS from the balancer only, and the subnet's network ACL, if one is used, admits it too. The server answers, the stateful security group lets the reply out, and the response returns along the same path. Each component in that walk stops a different problem: distance, overload, attacks in the request, and traffic from places that should never reach a server at all.

Common misconceptions

• "A network load balancer can send /api to one pool and /images to another." A layer 4 device never reads the path. Path routing needs layer 7.
• "Servers behind an application load balancer see each user's address as the source." They see the balancer's address; the client's address travels in X-Forwarded-For.
• "Security groups and network ACLs behave the same way." Security groups are stateful and allow replies automatically; network ACLs are stateless and need explicit rules for return traffic.
• "A CDN makes every response faster." Only cacheable responses are served from the edge. Personalized or uncacheable requests still go to the origin.

Pulling it together

• Network load balancers read addresses and ports at layer 4; application load balancers read requests at layer 7 and route by path, host and header.
• An application gateway is a layer 7 entry point for an application, usually with a web application firewall; Azure sells one under that name.
• Health checks remove failed targets; cross-zone balancing evens out uneven zones; X-Forwarded-For carries client addresses through layer 7 devices.
• CDNs serve fresh cached copies from edge locations; s-maxage and max-age set how long, and invalidation or versioned names handle urgent changes.
• Security groups, NSGs and Google's firewall rules are stateful; AWS network ACLs are stateless; managed firewalls add central inspection and intrusion prevention.

Sources

1. Amazon Web Services. (n.d.). How Elastic Load Balancing works, What is a Network Load Balancer?, What is an Application Load Balancer? and X-Forwarded headers. docs.aws.amazon.com (ELB); docs.aws.amazon.com (NLB); docs.aws.amazon.com (ALB); docs.aws.amazon.com (headers)
2. Microsoft. (n.d.). What is Azure Load Balancer?, What is Azure Application Gateway? and What is Azure Front Door? Microsoft Learn. learn.microsoft.com (Load Balancer); learn.microsoft.com (Application Gateway); learn.microsoft.com (Front Door)
3. Google Cloud. (n.d.). Cloud Load Balancing overview and Cloud NGFW firewall policies and rules. docs.cloud.google.com (load balancing); docs.cloud.google.com (firewalls)
4. Amazon Web Services. (n.d.). How CloudFront delivers content. Amazon CloudFront Developer Guide. docs.aws.amazon.com
5. Fielding, R., Nottingham, M., and Reschke, J. (Eds.). (2022). HTTP caching (RFC 9111). RFC Editor. rfc-editor.org
6. Amazon Web Services. (n.d.). Security groups for your VPC, Control subnet traffic with network access control lists and What is AWS Network Firewall? docs.aws.amazon.com (security groups); docs.aws.amazon.com (network ACLs); docs.aws.amazon.com (Network Firewall)
7. Microsoft. (n.d.). Network security groups and What is Azure Firewall? Microsoft Learn. learn.microsoft.com (NSGs); learn.microsoft.com (Azure Firewall)
8. HAProxy Technologies. (n.d.). HAProxy 3.2 configuration manual; Docker Library. (n.d.). haproxy official image; Traefik Labs. (n.d.). whoami. docs.haproxy.org; hub.docker.com; github.com

A belief worth testing: "the cluster has HA, so both database servers are safe"

Picture a cluster of eight virtualization hosts with high availability switched on. It runs a primary database virtual machine and its replica, and the team believes that no single failure can take both down. One afternoon a host loses its power supply, and both database VMs stop at the same moment. The HA service restarts them on surviving hosts a few minutes later, but the application is down until it does.

Trace the reasoning. HA in a virtualization cluster detects that a host has failed and restarts that host's VMs elsewhere; it does not keep them running, so a host failure still costs a restart. And nothing told the cluster's scheduler to keep the primary and the replica apart, so at some point both had been placed on the same host, a single failure domain. The missing control has a name: an anti-affinity rule, which says these VMs must run on different hosts. Its opposite, an affinity rule, keeps VMs together, for example an application server and the cache it talks to constantly.

This lesson works through several beliefs like that one, each plausible and each wrong, because each wrong turn teaches one part of CV0-004 objective 1.7: stand-alone hosts, clustering, cloning, host affinity, hardware pass-through, VM network types including overlays, and local, SAN and NAS storage.

Underneath: two kinds of hypervisor

NIST's guide to full virtualization, SP 800-125 from January 2011, describes two forms. In bare-metal virtualization, also called native, the hypervisor runs directly on the hardware with no host operating system. In hosted virtualization, the hypervisor runs on top of an ordinary operating system such as Windows, Linux or macOS. NIST notes that servers are most often virtualized with bare-metal hypervisors and desktops with hosted ones. It also makes a security point: a hosted hypervisor adds a whole host operating system, with its complexity and vulnerabilities, beneath your guests. Hosted products, too, rarely have access controls beyond whoever can log in to the host. VirtualBox, which you install in this lesson, is a hosted hypervisor; the hypervisors under public cloud instances are bare-metal designs.

A stand-alone host is one hypervisor machine working alone, like VirtualBox on your laptop: no shared storage, no partner hosts, and no way to restart its VMs elsewhere if it dies. A cluster joins hosts under shared management so that VMs can be placed, moved and restarted across them. Proxmox VE, an open-source virtualization platform, documents what its HA cluster needs: at least three nodes, so that a majority, or quorum, can always be formed, and shared storage reachable from every node. A node that loses quorum fences itself, stopping its services so two nodes never run the same VM at once.

Belief two: "HA will restart every VM somewhere else"

After the power-supply failure, one VM never restarted at all. Its virtual disk lived on the failed host's internal drives. HA could find a host with free CPU and memory, but no surviving host could reach the disk. That is the storage rule behind clustering: a VM can be restarted or moved only if its disks sit on storage every candidate host can reach.

The cloud makes the local row vivid. AWS says instance store data survives a reboot but does not persist if the instance is stopped, hibernated or terminated, and recommends it for buffers, caches, scratch data and data replicated across a fleet. Anything that must survive belongs on network block or file storage.

Worth holding on to: clustering restarts and moves VMs only as far as their storage reaches; local disks are fast and cheap, but they tie a VM to one host's fate.

Belief three: "clone the golden VM twenty times and we are done"

A clone is a copy of a VM. A full clone copies the virtual disks; a linked clone, which VirtualBox creates from a snapshot, shares the parent's disk and stores only its own changes, which is fast and saves space but ties the clone to the parent. Clones are how templates and cloud machine images work: build one well-configured VM, then stamp out copies.

The trap is identity. A copy of a configured machine copies everything that should have been unique. Windows administrators who clone a configured server without preparing it find the copies conflicting, because each carries the same computer security identifier. Microsoft's instructions are to generalize the image first with Sysprep, which removes computer-specific information such as installed drivers and the SID. Linux has the same issue in another file: systemd's documentation describes /etc/machine-id as the unique identity of a host, set at installation or first boot, and explains how images meant for many machines must leave it to be generated on first boot. The network layer needs uniqueness too. VirtualBox gives a clone new MAC addresses unless you ask it to keep them; keep them, attach the clones to one network, and switches, ARP and DHCP see one address in several places at once.

Belief four: "the GPU VM will live-migrate like the others"

Before patching a host, an administrator live-migrates its VMs to other hosts. All move except one that uses a GPU, which has to be shut down. The reason is hardware pass-through: the hypervisor hands a physical device, here a GPU, directly to one VM, using the processor's IOMMU support (Intel VT-d or AMD-Vi). The VM gets near-native performance, but the device belongs to that one host. NIST's guide states the trade plainly: direct device access can significantly increase speed, but it prevents some useful virtualization features, such as snapshots and moving guests while they are running. Proxmox's documentation adds that a passed-through device is no longer available to the host, and that VMs with passed-through devices cannot be migrated. Plan maintenance for such VMs as you would for physical servers.

Belief five: "VMs on different hosts share a network because they share a VLAN"

On a single host, VMs connect to virtual switches, and the VM network type decides who can reach whom. VirtualBox's manual lists the classic modes.

Across many hosts, a cluster or a cloud needs VMs in one virtual network to reach each other wherever they run. Stretching VLANs across a data center hits the limit you met in Lesson 5, 4094 VLAN IDs, so large platforms use overlay networks. Each host wraps VM traffic in a tunnel, such as VXLAN, and carries it across the physical network to the host where the destination VM lives. The VMs see one flat network; the physical network sees only traffic between hosts. That is the mechanism under every cloud VPC.

Affinity in the cloud: placement groups, fault domains and host affinity

You cannot pick a cloud host, but you can state placement intent, and the exam expects you to match each control to its purpose.

• Keep instances apart (anti-affinity). An AWS spread placement group strictly places a small group of instances on distinct underlying hardware to reduce correlated failures. Azure availability sets distribute VMs across fault domains and update domains for the same reason, though Microsoft notes they remain exposed to shared failures such as a data-center-wide network outage.
• Keep instances together (affinity). An AWS cluster placement group packs instances close together inside one Availability Zone for low-latency networking, the pattern tightly coupled computations need. Azure notes that availability sets offer lower VM-to-VM latency than zones because the VMs sit closer together.
• Keep an instance on one server (host affinity). On AWS Dedicated Hosts, setting an instance's affinity to Host means that an instance launched onto a specific host always restarts on that same host after a stop, which matters for software licensed per physical server.

Worth holding on to: affinity trades resilience for proximity; anti-affinity trades proximity for resilience. Decide which one each group of VMs needs, then write it down as a rule rather than trusting the scheduler.

Common misconceptions

• "HA means no downtime." Cluster HA restarts VMs after a host fails. The restart is an outage; only redundant VMs on separate hosts avoid it.
• "A clone is ready to use as soon as it boots." Without generalization, clones share identities such as Windows SIDs, Linux machine IDs and, if kept, MAC addresses.
• "Pass-through is just a faster driver." It hands a physical device to one VM, which ties that VM to its host and blocks live migration.
• "Instance store is the same as a cloud disk." It is local, temporary storage that is lost when the instance stops; network block volumes persist.

Summing up

• Bare-metal hypervisors run on the hardware and dominate servers; hosted hypervisors run on an operating system and suit desktops and labs.
• Clusters restart and move VMs across hosts, but only with shared SAN or NAS storage, and an HA restart is still downtime.
• Anti-affinity keeps redundant VMs apart; affinity keeps chatty VMs together; host affinity pins an instance to one physical server.
• Generalize before cloning (Sysprep, machine-id) and give clones new MAC addresses.
• Pass-through gives near-native device performance at the cost of migration and snapshots.
• VM networks (NAT, bridged, host-only, internal) shape reachability on one host; overlay networks such as VXLAN connect VMs across hosts.

Sources

1. Scarfone, K., Souppaya, M., and Hoffman, P. (2011). Guide to security for full virtualization technologies (NIST Special Publication 800-125). National Institute of Standards and Technology. csrc.nist.gov
2. Proxmox Server Solutions. (n.d.). High availability and PCI passthrough. Proxmox VE wiki. pve.proxmox.com (HA); pve.proxmox.com (passthrough)
3. Oracle. (n.d.). Oracle VirtualBox user guide: virtual networking and VBoxManage clonevm. virtualbox.org (networking); virtualbox.org (VBoxManage)
4. Microsoft. (n.d.). Sysprep (generalize) a Windows installation. Microsoft Learn. learn.microsoft.com
5. systemd project. (n.d.). machine-id [Manual page]. freedesktop.org
6. Amazon Web Services. (n.d.). Placement groups, Dedicated Host auto-placement and host affinity and Instance store temporary block storage. Amazon EC2 User Guide. docs.aws.amazon.com (placement); docs.aws.amazon.com (host affinity); docs.aws.amazon.com (instance store)
7. Microsoft. (n.d.). Availability sets overview. Microsoft Learn. learn.microsoft.com
8. Mahalingam, M., and others. (2014). Virtual eXtensible Local Area Network (VXLAN) (RFC 7348). RFC Editor. rfc-editor.org

One artifact from laptop to production

NIST's Application Container Security Guide, Special Publication 800-190 from September 2017, describes a problem every operations team knows. Without containers, an application is installed in production by scripts run on the server. The tests in the build pipeline and on developers' laptops never test the real production artifact, only an approximation that drifts over time: the "it works on my machine" problem. With containers, NIST explains, the build installs the application into an image, an immutable snapshot of everything the application needs above the kernel, and the developer, the build system and production all run that same image.

The same guide states the price. Containers on a host share that host's operating system kernel, which is what makes them small and quick to start. It also makes them OS-family specific: a Linux host can run only Linux containers, and a Windows host only Windows containers. It also means, in NIST's words, that containers do not offer as clear and concrete a security boundary as a virtual machine, because they share one kernel and can run with varying privileges.

This lesson builds one container end to end, then changes a single input to show how data disappears. It covers CV0-004 objective 1.6 for stand-alone containers, container networking and port mapping, persistent and ephemeral storage, and image registries, and it introduces Docker, one of the DevOps tools in objective 5.4. Workload orchestration, the rest of 1.6, is Lesson 10.

Step 1: build an image

An image is built from a Dockerfile, a short list of instructions. For a one-page website, three lines are enough.

1. FROM nginx:alpine starts from a published image that already contains a web server.
2. COPY index.html /usr/share/nginx/html/index.html adds your page where that image's documentation says the server looks for content.
3. EXPOSE 80 records that the application listens on port 80.

Running docker build -t hello:1.0 . turns those lines into an image named hello with the tag 1.0. Each instruction that changes files adds a read-only layer on top of the base image's layers, and docker image history hello:1.0 lists them. Read the third line carefully, because the exam does: Docker's reference says the EXPOSE instruction does not actually publish the port; it functions as documentation between the person who builds the image and the person who runs it.

Step 2: run it stand-alone and map a port

docker run -d --name hello -p 127.0.0.1:8080:80 hello:1.0 starts a stand-alone container: one container, on one host, managed by that host's container engine alone. The container gets a thin writable layer on top of the image's read-only layers, and its own network namespace, so port 80 inside it is not port 80 on your computer.

The -p flag is port mapping, which Docker calls publishing. It is written host first, container second: requests to port 8080 on the host are forwarded to port 80 in the container. Without -p, nothing outside the container's networks can reach it, whatever the Dockerfile's EXPOSE line says. Docker's documentation adds a warning worth memorizing: publishing container ports is insecure by default, because a published port is reachable on the host's network addresses, so anyone who can reach your machine can reach the container. Including the loopback address, as in 127.0.0.1:8080:80, limits access to the host itself.

Two containers can both listen on port 80 inside, because each has its own namespace, but one host address and port can be published only once. A second container asking for 127.0.0.1:8080 fails to start until you map it to another host port, such as 8081.

So what? EXPOSE documents, -p publishes, the host port comes first, and a published port is open to every network your host is on unless you name an address.

Step 3: tag, push and pull by digest

To run the image anywhere else, you push it to a registry, a server that stores images and hands them out. Docker Hub is the best-known public registry, and each provider runs a private one: Amazon Elastic Container Registry, Azure Container Registry and Google's Artifact Registry. All of them speak the same protocol, the Open Container Initiative's Distribution Specification, which defines an API to standardize the distribution of content. In its vocabulary a repository holds an image's versions, each blob is addressed by a digest computed from a cryptographic hash of its content, and a tag is a human-readable name pointing at content.

You can run the reference registry yourself. docker run -d -p 127.0.0.1:5000:5000 --name registry registry:3 starts it; docker tag hello:1.0 localhost:5000/hello:1.0 gives your image a name that includes the registry's address; docker push localhost:5000/hello:1.0 uploads its layers. On another host, a pull of that name would download exactly those layers.

The difference between a tag and a digest matters in production. A tag can be moved: push a new image as hello:1.0, or more commonly as latest or stable, and every future pull gets the new content. A digest cannot move, which is why Docker's documentation describes pulling by digest as pulling by an immutable identifier. Deployments that must be repeatable reference images by digest, or by tags the team treats as never-reused.

Step 4: give the data somewhere to live

Data written into a container's writable layer belongs to that container. Docker's storage documentation says such data does not persist when the container no longer exists. NIST's guide makes the design rule explicit: persistence should come from external data stores or persistent volumes. Then, when the next version of an application replaces the running containers, all the data is still there.

The objective's two words map onto that table. Persistent storage outlives the container; ephemeral storage lives and dies with it. Kubernetes, in Lesson 10, offers the same pair as persistent volumes and ephemeral volumes.

The second run: a routine upgrade that deletes the data

Now change one input. A team runs a database in a container with no volume, and its data directory sits in the writable layer. Everything works for months. Then they upgrade in the usual container way: remove the old container and start a new one from the newer image. The new container starts with an empty writable layer, so the database is empty. Nothing malfunctioned; the design never said where the data should live.

Run the same upgrade with a named volume mounted at the database's data directory and the result flips. The old container is removed, the volume remains, the new container mounts it, and the data is all there. The pattern generalizes: containers should be disposable, and anything that must outlive one belongs in a volume or an external service.

Common misconceptions

• "EXPOSE 80 makes the site reachable." EXPOSE is documentation. Only publishing with -p maps a host port to the container.
• "A container is a lightweight VM with its own kernel." Containers share the host's kernel, which is why a Linux host cannot run Windows containers and why the isolation boundary is weaker than a VM's.
• "The latest tag always means the newest release." It is just a tag name, and tags can be repointed at any image. Pin by digest when it matters.
• "Publishing a port on my laptop only exposes it to me." By default it is reachable on all the host's addresses; bind to 127.0.0.1 for local-only access.

Looking back

• An image is an immutable, layered package of an application and its userspace; a container is a running instance with a thin writable layer, sharing the host's kernel.
• A stand-alone container is managed by one engine on one host, with no rescheduling if that host fails.
• Port mapping publishes host:container ports; EXPOSE only documents; publishing is open on all host addresses unless you bind to one.
• Registries store images under the OCI Distribution protocol; tags can move, digests cannot.
• Volumes and bind mounts persist; the writable layer and tmpfs are ephemeral. Design upgrades so containers can be replaced without losing data.

Sources

1. Souppaya, M., Morello, J., and Scarfone, K. (2017). Application container security guide (NIST Special Publication 800-190). National Institute of Standards and Technology. csrc.nist.gov
2. Docker. (n.d.). Dockerfile reference (EXPOSE). Docker Docs. docs.docker.com
3. Docker. (n.d.). Port publishing and mapping. Docker Docs. docs.docker.com
4. Docker. (n.d.). Storage (volumes, bind mounts and tmpfs mounts). Docker Docs. docs.docker.com
5. Docker. (n.d.). docker image pull (pull an image by digest). Docker Docs. docs.docker.com
6. Open Container Initiative. (n.d.). OCI distribution specification. github.com
7. CNCF Distribution project. (n.d.). Distribution registry [Official image documentation]. distribution.github.io
8. Google Cloud. (n.d.). Compare AWS and Azure services to Google Cloud (container registries). docs.cloud.google.com

Thirty containers, five hosts and a pager

Lesson 9 ended with a stand-alone container host that crashed overnight while two other hosts sat idle. Now scale the problem up. An application needs thirty containers spread across five hosts. Hosts fail now and then, containers crash, a new version ships every week, and the other services that call this application need to find its containers even though they keep moving. Doing that by hand means a person with a pager restarting containers at 3 AM. What, exactly, would software have to do to take that person's place?

It would need to do five jobs: decide which host runs each container; notice failures and replace what failed; give callers a stable address for a set of containers that keeps changing; replace old versions with new ones without an outage; and attach storage that outlives any one container. Those five jobs are workload orchestration, the remaining part of CV0-004 objective 1.6, and Kubernetes, one of the DevOps tools in objective 5.4, is the orchestrator you are most likely to meet. This lesson works through the jobs one at a time on the kind cluster from Lesson 1.

The idea underneath: desired state and control loops

The Kubernetes documentation describes it as a portable, extensible, open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation, building on more than fifteen years of Google's experience running production workloads. The key word is declarative. You do not tell Kubernetes to start three containers; you declare that three should exist, and Kubernetes keeps making it true.

The mechanism is the control loop. Kubernetes' own analogy is a thermostat: you set the desired temperature, the room has a current temperature, and the thermostat switches equipment on or off to bring the current state closer to the desired one. In Kubernetes, controllers are control loops that watch the cluster's state, compare it with the desired state recorded in each object's spec, and act to close the gap. Every job in this lesson is a controller doing exactly that.

Job one: placing work

Kubernetes never schedules a bare container. Its smallest deployable unit is the Pod: one or more containers that share storage and network resources and always run together on the same node. The scheduler places each new Pod on a node that has room for what the Pod requests. If no node can fit it, the Pod waits. Kubernetes' debugging guide puts it plainly: a Pod stuck in Pending cannot be scheduled onto a node, and generally that is because of insufficient resources of one type or another. Asking for 64 CPUs on a cluster of 4-CPU nodes produces a Pending Pod forever, a resource-allocation fault you will meet again in Lesson 33.

Job two: healing

You rarely create Pods directly. A Deployment declares how many identical Pods you want and which image they run; it manages a ReplicaSet, and the ReplicaSet's controller keeps the number of Pods equal to the number declared. Delete one of three Pods and the count drops to two; the controller notices within moments and creates a replacement. The overview page lists this as self-healing: Kubernetes restarts containers that fail, replaces containers, and kills containers that do not respond to your health checks.

Whole nodes fail too. When one stops responding, Kubernetes marks it, and after the default 300-second toleration you met in Lesson 4, its Pods are evicted and the Deployment's controller creates replacements on healthy nodes. That is the job the stand-alone host could not do.

The core of it: you declare the result, not the steps. A controller compares what exists with what you declared and acts, forever, which is why deleting a Pod managed by a Deployment does not delete the application.

Job three: finding Pods that keep moving

Every replacement Pod gets a new IP address, so callers cannot use Pod addresses. A Service solves this: it gives a set of Pods one stable virtual address and DNS name, and it finds its Pods by label. Labels are key-value tags, such as app=web, and a Service's selector says which labels its Pods must carry. Kubernetes keeps the list of matching Pod addresses, the endpoints, up to date as Pods come and go.

Port mapping appears again here. A Service can map any incoming port to a different targetPort on its Pods, so a Service on port 80 can forward to containers listening on 8080, much as -p did for a single container in Lesson 9. For testing, kubectl port-forward forwards a port on your own computer to a Service or Pod without exposing anything.

The commonest Service fault follows from the design. If the selector says app=web and the Pods are labelled app=frontend, the Service has no endpoints: it exists, it has an address, and every connection to it fails. Checking labels against selectors is the first step whenever a Service answers nothing.

Job four: rolling out new versions

Changing a Deployment's image starts a controlled replacement: new Pods are created and old ones removed in steps, so the application keeps serving throughout, and a bad version can be rolled back. Lesson 20 compares that rolling strategy with blue-green and canary deployments, which Kubernetes can also express.

Job five: storage that outlives a Pod

Containers in a Pod can share an emptyDir volume, created empty when the Pod is assigned to a node. It is ephemeral storage: the documentation states that when a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently. Use it for scratch space and caches.

For data that must survive, Kubernetes separates supply from demand. A PersistentVolume is a piece of storage in the cluster; a PersistentVolumeClaim is a request for storage by a user, much as a Pod requests CPU and memory. A StorageClass lets claims be met automatically by creating a new volume on demand. Your kind cluster ships with a default class named standard, backed by a local-path provisioner that carves volumes out of the node's disk; in a managed cloud cluster, a storage class can instead create the provider's network block volumes. Because the claim, not the Pod, owns the volume, a replacement Pod that mounts the same claim finds the data waiting.

Running Kubernetes yourself, or renting it

The control plane is the hard part to operate: it must be highly available, backed up, upgraded and secured. All three major providers will run it for you, as Amazon Elastic Kubernetes Service, Azure Kubernetes Service and Google Kubernetes Engine. AWS describes EKS standard as managing the Kubernetes control plane for you, including the components that schedule workloads and store control plane information. You still own your workloads, their configuration and access to them, the shared responsibility pattern from Lesson 2 applied to a platform. Running Kubernetes yourself, as you do with kind, gives full control and full responsibility; it is the right way to learn and, for most organizations, the wrong way to run production.

Common misconceptions

• "Kubernetes schedules containers." It schedules Pods, the smallest deployable unit, each holding one or more containers that share a network and storage.
• "Deleting a Pod stops the application." If a Deployment manages it, the controller creates a replacement to restore the declared count.
• "A Service is a load balancer appliance." A Service is a stable address and a label selector. Only the LoadBalancer type asks the provider for an external load balancer.
• "emptyDir is persistent storage." Its data is deleted permanently when the Pod leaves its node. Persistence needs a PersistentVolumeClaim.

What to remember

• Orchestration means placing, healing, finding, updating and storing for many containers on many hosts; Kubernetes does it with controllers that drive current state toward declared desired state.
• The control plane (API server, etcd, scheduler, controller manager) decides; kubelets and container runtimes on each node do.
• Pods are scheduled by requested resources and wait in Pending when nothing fits; Deployments keep the declared number running.
• Services give moving Pods one address, found by label selectors; ClusterIP is internal, NodePort uses 30000 to 32767, LoadBalancer asks the provider.
• emptyDir is ephemeral; PersistentVolumeClaims bind persistent storage that outlives Pods.
• EKS, AKS and GKE run the control plane for you; your workloads and their security stay yours.

Sources

1. The Kubernetes Authors. (n.d.). Overview and Kubernetes components. Kubernetes documentation. kubernetes.io (overview); kubernetes.io (components)
2. The Kubernetes Authors. (n.d.). Controllers and Pods. Kubernetes documentation. kubernetes.io (controllers); kubernetes.io (Pods)
3. The Kubernetes Authors. (n.d.). Service. Kubernetes documentation. kubernetes.io
4. The Kubernetes Authors. (n.d.). Volumes and Persistent volumes. Kubernetes documentation. kubernetes.io (volumes); kubernetes.io (persistent volumes)
5. The Kubernetes Authors. (n.d.). Debug Pods. Kubernetes documentation. kubernetes.io
6. Kubernetes SIGs. (n.d.). kind source: default storage class manifest (const_storage.go). github.com
7. Amazon Web Services. (n.d.). What is Amazon EKS? Amazon EKS User Guide. docs.aws.amazon.com
8. Google Cloud. (n.d.). Compare AWS and Azure services to Google Cloud (managed Kubernetes). docs.cloud.google.com

Two articles, fifteen months apart

On 25 March 2014 James Lewis and Martin Fowler published an article that gave a loose movement its name. They described microservices as an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP API, built around business capabilities and independently deployable. On 3 June 2015 Fowler published a shorter piece with a warning in its title, MonolithFirst. Almost all the successful microservice stories he had heard, he wrote, started with a monolith that got too big and was broken up, and almost every system he had heard of that was built as microservices from scratch had ended up in serious trouble.

Both pieces are right about something, which is why the argument has not ended. This lesson sets out each position with its evidence, then the design patterns that both sides rely on. It covers CV0-004 objective 1.5, the purpose of cloud-native design concepts: cloud-provided managed services, microservices, loosely coupled architecture, fan-out and service discovery.

The case for microservices

NIST's guide to securing microservices, Special Publication 800-204 from August 2019 by Ramaswamy Chandramouli, lists the incentives that organizations give for the style. Each service typically implements a single business function, so its codebase is small and less complex, which speeds development, testing and deployment. Teams can work independently because the services are loosely coupled. Each component can be scaled on its own and run on a platform suited to it. NIST also notes that microservice applications are often identified as cloud-native, combining service-based architecture, API-driven communication, container infrastructure and DevOps practices.

Make that concrete with an online bookstore. Search needs ten times its usual capacity in December, while checkout barely changes. As separate services, search scales alone; the checkout team ships a fix on Tuesday without waiting for the search team's release; a bug that crashes recommendations leaves checkout running. Each of those benefits comes from independence: separate processes, separate deployments, separate failure.

The case for starting with a monolith

Fowler's 2015 argument rests on what he calls the microservice premium: the extra cost of running many services pays off only for systems complex enough to need it, and slows a team down when they are not. Two reasons stand out. First, you rarely know the right service boundaries at the start, and moving a boundary between services is far harder than moving one between modules in a single codebase. Second, distribution itself costs something. NIST's guide is blunt here: network security, reliability and latency become critical factors because every transaction in a microservices system involves sending messages across a network. A request that used to be a function call becomes several network calls, each of which can be slow, fail or be attacked, and the system now needs service discovery, load balancing, failure detection and monitoring between its parts.

So the second position is not "never microservices". It is: start with a well-structured, modular monolith, learn where the real boundaries are, and split pieces out when a specific pressure, such as a component with a different scaling profile or a team that must deploy independently, justifies the premium.

Bottom line: microservices buy independence at the price of distribution. Pay it when you need the independence, and not before.

Loose coupling: what both sides need

Whichever side you take, the parts of a system should depend on each other as little as possible. AWS's Well-Architected reliability guidance, best practice REL04-BP02, explains why. Decoupling dependencies through queues, streams and workflows minimizes the impact of changes or failures, because it isolates one component's behaviour from those that depend on it. In tightly coupled systems a change to one component forces changes in others; with loose coupling, dependent components need to know only the versioned, published interface. A queue between an order service and a warehouse service lets the warehouse fall behind or restart without the order service noticing, because orders simply wait in the queue. A modular monolith can be loosely coupled inside, through clean module interfaces; a set of microservices that call each other synchronously in long chains can be tightly coupled despite running in separate processes.

Fan-out: one event, many consumers

When an order is placed, three things must happen: charge the card, email a receipt and update analytics. The order service should not call all three in turn, waiting on each and failing if one is down. Instead it publishes one event to a topic, and the messaging service delivers a copy to every subscriber, each of which processes it at its own pace. That is fan-out. AWS's documentation describes the classic form, an SNS topic fanning out to several SQS queues so that messages are processed in parallel and asynchronously, and the other providers offer the same pattern through Google's Pub/Sub and Azure Service Bus topics.

Do not confuse fan-out with load balancing. A load balancer gives each request to one of many targets so they share the work. Fan-out gives each message to every subscriber so each can do a different job with it.

Service discovery: finding things that move

NIST explains why microservices need it: there are many services, many instances of each, and their locations change dynamically as instances start, stop and move. Hard-coding addresses fails the first time an instance is replaced. Service discovery keeps a current map from names to locations. In Kubernetes, which you met in Lesson 10, it is built in: Kubernetes creates DNS records for Services, so callers use consistent names such as orders.shop.svc followed by the cluster domain, which resolve to the Service's stable address. Outside Kubernetes, AWS Cloud Map is a managed service that maps logical names to the backend services and resources an application depends on, discoverable through DNS or an API call; tools such as HashiCorp Consul do the same across platforms.

Cloud-provided managed services: renting the undifferentiated parts

A message broker, a database, a service registry and a load balancer are all things a team could install and run itself on virtual machines. The cloud-native habit is to ask whether it should. The AWS Well-Architected Framework's cost guidance puts it as a design principle: stop spending money on undifferentiated heavy lifting, because managed services remove the operational burden of managing the operating systems and applications underneath. A team that spends hours a month patching and scaling its own broker could use a managed messaging service and spend those hours on its product. The trade-offs are the ones from Lesson 2: less control, the provider's limits and quotas, and a closer tie to one provider's interfaces, which matters again in Lesson 16 when the subject is vendor lock-in.

What would settle it for your system

The dispute is not settled by opinion but by facts about your situation. Count the teams that need to release independently: one team gains little from independent deployment. Look for components with genuinely different load profiles or reliability needs. Check whether you already have automated deployment, centralized logging and distributed tracing, because a distributed system without them is very hard to debug, as Lesson 21 shows. And set a latency budget, then count the network calls a request would make. If most answers point to independence, split along the boundaries your monolith has already revealed; if not, keep the monolith modular and loosely coupled, on managed services where they fit.

Bottom line: cloud-native is less about the number of services than about loose coupling, automation and letting the provider run the undifferentiated parts.

Common misconceptions

• "Cloud-native means microservices." Managed services, loose coupling and automation matter more; a modular monolith on managed services can be thoroughly cloud-native.
• "Microservices are always more scalable and cheaper." They scale parts independently, but every network hop adds latency, failure modes and security work, which NIST lists as critical challenges.
• "Fan-out is a kind of load balancing." Load balancing sends each request to one target; fan-out sends each message to every subscriber.
• "Service discovery is a list of IP addresses kept in a config file." Discovery is dynamic: instances register and disappear, and names resolve to whatever is running now.

The takeaway

• Lewis and Fowler (2014) defined microservices as small, independently deployable services in their own processes; Fowler (2015) warned that successful ones usually start as monoliths.
• NIST SP 800-204 lists the incentives, small codebases, team independence and independent scaling, and the costs: network security, reliability and latency on every transaction.
• Loose coupling through queues, streams and published interfaces limits how far a change or failure spreads.
• Fan-out delivers one event to every subscriber; service discovery maps names to moving instances; managed services remove undifferentiated operational work at the price of control.

Sources

1. Lewis, J., and Fowler, M. (2014, March 25). Microservices. martinfowler.com. martinfowler.com
2. Fowler, M. (2015, June 3). MonolithFirst. martinfowler.com. martinfowler.com
3. Chandramouli, R. (2019). Security strategies for microservices-based application systems (NIST Special Publication 800-204). National Institute of Standards and Technology. csrc.nist.gov
4. Amazon Web Services. (n.d.). REL04-BP02 Implement loosely coupled dependencies. AWS Well-Architected Framework, Reliability Pillar. docs.aws.amazon.com
5. Amazon Web Services. (n.d.). Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processing. docs.aws.amazon.com
6. The Kubernetes Authors. (n.d.). DNS for Services and Pods. Kubernetes documentation. kubernetes.io
7. Amazon Web Services. (n.d.). What is AWS Cloud Map? docs.aws.amazon.com
8. Amazon Web Services. (n.d.). Design principles (cost optimization). AWS Well-Architected Framework. docs.aws.amazon.com
9. Eclipse Foundation. (n.d.). Migrating from 1.x to 2.0. Eclipse Mosquitto documentation. mosquitto.org

A weather record from 1763, by name

NOAA, the US National Oceanic and Atmospheric Administration, publishes its Global Historical Climatology Network daily records in a public Amazon S3 bucket called noaa-ghcn-pds. Ask it for a list of three objects with a single HTTPS request, as the lab in this lesson does, and it answers with a short XML document. On 28 October 2026 the first entry was a key, csv.gz/1763.csv.gz, with a size of 3,358 bytes, a last-modified date in September 2022 and a storage class of STANDARD. There was no directory to open and no disk to mount: a name, some metadata and the bytes behind them.

That is object storage, one of three ways the cloud stores data. This lesson compares them, then compares the disks and the temperature tiers underneath, and finishes with what each choice costs. It covers CV0-004 objective 1.4: tiered storage (hot, warm, cold and archive), disk types (SSD and HDD), storage types (object, block and file), and their performance and cost implications.

Three ways to store data

Read the object column with the 1763 record in mind. AWS describes S3 as storing data as objects within buckets: an object is a file and any metadata that describes it, a bucket is a container for objects, and each object has a key that uniquely identifies it in the bucket. The slash in csv.gz/1763.csv.gz is just a character in that key. Consoles draw such prefixes as folders for convenience, but the namespace is flat, and an upload replaces an object as a whole rather than editing it in place. File storage, by contrast, is a real hierarchy shared by many clients; Microsoft notes that an Azure file share can be mounted concurrently over SMB and NFS from Windows, Linux and macOS. Block storage is the lowest level: a volume your server formats and owns, which is why databases and boot disks live there.

Key idea: block storage gives one server a disk, file storage gives many clients a shared folder tree, and object storage gives any authorized client named objects over HTTPS.

Disk types and performance

Three numbers describe storage performance, and a workload usually cares about one more than the others. IOPS is the number of input and output operations per second, which matters for many small random reads and writes, such as a database's. Throughput is the volume of data moved per second, which matters for large sequential reads, such as a backup or a video. Latency is how long one operation takes.

The disk technology underneath sets the trade-off. AWS's description of its volume families says SSD-backed volumes are optimized for transactional workloads with frequent small reads and writes, where the dominant attribute is IOPS, while HDD-backed volumes are optimized for large streaming workloads, where the dominant attribute is throughput. AWS's Throughput Optimized HDD (st1) and Cold HDD (sc1) volumes cannot be boot volumes, and its Cold HDD averages about 100 IOPS, a number any SSD volume dwarfs. Azure's managed disks span the same range, from Ultra Disk and Premium SSD v2 down through Premium SSD and Standard SSD to Standard HDD, and Microsoft has announced that Standard HDDs can no longer be used as operating system disks after 8 September 2028. The pattern is the same everywhere: SSD for IOPS and boot disks, HDD for cheap sequential capacity.

Tiers: hot, warm, cold and archive

Object storage adds a second dial, the access tier. Colder tiers cost less to store data in, but more to read it, and some require a minimum stay. The providers' names do not line up perfectly, so treat this mapping to CompTIA's four words as approximate.

The last row holds the trap that scenario questions like. AWS's deepest tier needs hours to restore an object, and Azure's archive tier is offline: an archived blob cannot be read until it is rehydrated to an online tier. Google's Archive class, by its own description, serves data in milliseconds, not hours or days, but asks for a full year's minimum. An auditor who needs a document within the hour is fine with Google's Archive, and waits most of a day or longer with AWS Deep Archive.

Some services move data between tiers for you. S3 Intelligent-Tiering monitors access and moves each object between tiers automatically, for a small monitoring fee and with no retrieval fees. Otherwise, lifecycle rules do it on a schedule: for example, move objects to a cool tier after 30 days and to archive after 90.

Cost: three meters, not one

A storage bill has three parts, and colder tiers move money between them rather than simply removing it.

1. Capacity: the price per gigabyte per month, which falls as the tier gets colder.
2. Access: charges for requests and for retrieving data, which rise as the tier gets colder. Microsoft states it plainly: the hot tier has the highest storage costs but the lowest access costs, and each colder tier lowers storage costs and raises access costs.
3. Minimum duration: AWS says that if you delete, overwrite or transition an object in a Glacier class before its minimum, you are charged for the remainder of that duration; Azure applies early-deletion charges to its cool, cold and archive tiers in the same spirit.

Work an example without prices, which change. A company keeps application logs for seven years for audit. Logs are read heavily for two weeks and almost never afterward. Keeping seven years of logs in a hot tier pays the highest capacity price for data nobody reads. Writing everything straight to archive pays retrieval charges and waits hours every time someone investigates last week's incident. The right design follows the data's temperature: hot for the first weeks, a cooler tier for the following months, archive for the remaining years, with each move timed to fall after the tier's minimum. Now change one input: if the company deleted logs after 45 days, a tier with a 90-day minimum would charge for days it never used, and staying warm would be cheaper.

Common misconceptions

• "Object storage has folders." Keys may contain slashes, and consoles draw them as folders, but the namespace is flat: a key is just a name.
• "Archive is simply a cheaper hot tier." Archive trades capacity price for retrieval time, retrieval fees and minimum durations; at AWS and Azure, reading archived data takes hours.
• "A block volume can be shared by many servers like a file share." A block volume normally belongs to one server, which formats it. Shared access by many clients is what file storage is for.
• "SSD is always the right choice." For large sequential throughput on a budget, HDD volumes are designed for exactly that job, as long as they are not boot volumes.

Recap

• Block storage is a raw volume for one server; file storage is a shared hierarchy over NFS or SMB; object storage is named objects with metadata in a flat bucket, reached over HTTPS.
• IOPS, throughput and latency describe performance; SSDs lead on IOPS and boot disks, HDDs on cheap sequential throughput.
• Hot, warm, cold and archive tiers lower capacity cost while raising access cost and minimum durations; AWS Deep Archive restores take 9 to 48 hours, Azure's archive is offline, Google's Archive answers in milliseconds.
• Plan with all three meters, capacity, access and minimum duration, and use lifecycle rules or automatic tiering to follow the data's temperature.

Sources

1. Amazon Web Services. (n.d.). What is Amazon S3?, Understanding and managing Amazon S3 storage classes and Understanding S3 Glacier storage classes. Amazon S3 User Guide. docs.aws.amazon.com (S3); docs.aws.amazon.com (classes); docs.aws.amazon.com (Glacier)
2. Amazon Web Services. (n.d.). Amazon EBS volume types. Amazon EBS User Guide. docs.aws.amazon.com
3. Microsoft. (n.d.). Access tiers for blob data and Blob rehydration from the archive tier. Microsoft Learn. learn.microsoft.com (tiers); learn.microsoft.com (rehydration)
4. Microsoft. (n.d.). Azure managed disk types and What is Azure Files? Microsoft Learn. learn.microsoft.com (disks); learn.microsoft.com (Files)
5. Google Cloud. (n.d.). Storage classes. Cloud Storage documentation. docs.cloud.google.com
6. National Oceanic and Atmospheric Administration. (n.d.). Global Historical Climatology Network daily [Public bucket noaa-ghcn-pds, listing retrieved 28 October 2026]. noaa-ghcn-pds.s3.amazonaws.com
7. Google Cloud. (n.d.). Compare AWS and Azure services to Google Cloud (storage services). docs.cloud.google.com

A wrong answer that sounds modern

PostgreSQL's own tutorial explains transactions with a bank transfer: money leaves Alice's account and arrives in Bob's, and the documentation insists that the two updates form a single, all-or-nothing operation. If anything fails partway, neither step affects the database, and while the transfer is in progress no other transaction may see one half without the other. Hold that example next to a sentence often heard in planning meetings: "Non-relational databases are faster and scale better, so we should use one for the new order system."

That sentence is the plausible wrong answer this lesson takes apart, along with three others about consistency and managed services. Each wrong turn teaches part of CV0-004 objective 1.9: relational and non-relational database types, and self-managed and provider-managed deployment. For writing SQL itself, Databases & SQL (CS 340) is the place to go deeper; here the question is which kind of database, run by whom.

Tracing the first error: what the order system actually needs

Start from the workload, not the label. Placing an order writes an order record, several line items, a payment and an inventory decrement, and all four must happen together or not at all. Finance runs reports that join orders to customers, products and refunds in ways nobody predicted when the system was built. Both requirements point the same way. Microsoft's guide to data store models describes relational database management systems as organizing data into normalized tables with a schema defined before data is written, enforcing integrity, and supporting ACID transactions and rich SQL queries. ACID stands for atomicity, consistency, isolation and durability, the properties PostgreSQL's bank transfer illustrates.

The claim about scale is not false, just misapplied. The same guide notes that horizontal scaling of a relational database generally requires sharding or partitioning, which is real work. Non-relational stores are designed around known access patterns and often scale out more easily, but they give up some of what relational systems provide. Microsoft notes, for example, that most key-value stores support only simple query, insert and delete operations, and that to change part of a value the application must overwrite the whole value. For an order system with transactions and ad hoc reports, a relational database is the stronger fit, and it can scale a long way before sharding is needed.

The point: choose a database by the shape of the data and the questions you will ask of it. "Relational or not" is a decision about transactions, joins and access patterns, not about which technology sounds newer.

The second error: "the database lost my write"

A developer writes an item to Amazon DynamoDB, a serverless, fully managed NoSQL database that AWS describes as delivering single-digit millisecond performance at any scale, and immediately reads it back. Usually the new value appears; occasionally the old one does. The bug report says the database lost the write. It did not. DynamoDB's documentation states that eventually consistent reads are the default for all read operations, so a read soon after a write may return the older item. When the application needs the latest value, it can ask for a strongly consistent read, which costs twice as much: eventually consistent reads are half the price.

The general lesson is that distributed databases make a consistency choice, and many default to the cheaper, faster, eventually consistent option. Before blaming the database, check which read model the code asked for.

The third error: "managed means everything is handled"

Provider-managed databases are a real reduction in work. AWS describes Amazon RDS as responsible for most management tasks, managing backups, software patching and automatic failure detection, and for hosting the software components and infrastructure of database instances. But the same page draws a line: you remain responsible for query tuning, and the shared responsibility model from Lesson 2 still leaves data, schema, users and network access with you.

Some protections are also options rather than defaults you can assume. Running a database in several availability zones, with a standby and automatic failover, is a Multi-AZ deployment that you choose when you create or modify the instance. Automated backups follow a retention period that you set, and point-in-time recovery works within it. A team that never chose Multi-AZ has a managed database in one zone, and Lesson 4 showed what a zone failure does to that.

The fourth error: "running it ourselves is cheaper"

Self-managed means installing a database engine on virtual machines, or in containers, and operating it yourself. It gives full control: any engine, any version, any extension, any configuration. The comparison that matters is not the instance price alone but the work.

Count the hours in the left column and the comparison often reverses. Self-managed still wins when you need something the managed service does not offer, such as an unsupported extension or version, or when licensing or regulation requires control of the host.

The point: managed databases remove the undifferentiated work of patching, backups and failure detection, but resilience options must be chosen, and data, schema, queries and access are always yours.

Common misconceptions

• "NoSQL is simply the faster, more modern relational database." It is a family of different models built for particular access patterns, usually giving up joins or general-purpose transactions in exchange.
• "Relational databases cannot scale." They scale up and out for reads a long way; scaling writes horizontally requires sharding or partitioning, which is harder but done routinely.
• "A stale read means data loss." With eventually consistent reads, as DynamoDB uses by default, a recent write may not appear yet. Ask for a strongly consistent read when it matters.
• "A managed database is automatically highly available and backed up forever." Multi-AZ and backup retention are settings you choose.

What to carry forward

• Relational databases give normalized tables, ACID transactions and SQL; document, key-value, column-family and graph stores each fit a particular data shape and access pattern.
• Transactions are all-or-nothing; eventually consistent reads may lag a write, and strongly consistent reads cost more.
• Provider-managed databases take over patching, backups and failure detection; you still own data, schema, queries, access, and the choice of options such as Multi-AZ and retention.
• Self-managed databases give full control at the price of all the operational work.

Sources

1. The PostgreSQL Global Development Group. (n.d.). Concepts and Transactions. PostgreSQL documentation, tutorial. postgresql.org (concepts); postgresql.org (transactions)
2. Microsoft. (n.d.). Understand data store models. Azure Architecture Center. learn.microsoft.com
3. Amazon Web Services. (n.d.). What is Amazon DynamoDB? and DynamoDB read consistency. Amazon DynamoDB Developer Guide. docs.aws.amazon.com (overview); docs.aws.amazon.com (consistency)
4. Amazon Web Services. (n.d.). What is Amazon Relational Database Service (Amazon RDS)? and Introduction to backups. Amazon RDS User Guide. docs.aws.amazon.com (RDS); docs.aws.amazon.com (backups)
5. Google Cloud. (n.d.). Compare AWS and Azure services to Google Cloud (databases). docs.cloud.google.com
6. Docker Library and Valkey project. (n.d.). postgres official image and Valkey. hub.docker.com; valkey.io

By the second, or up to 72 percent off

AWS describes its On-Demand Instances in one sentence: you pay for compute capacity by the hour or second, with a minimum of 60 seconds and no long-term commitment. Microsoft describes Azure Reservations in another: commit to a one-year or three-year plan and reduce resource costs by up to 72 percent from pay-as-you-go prices. Between those sentences sits most of what cloud cost management is about. Paying only for what you use is the cloud's promise, and committing to what you will use is how organizations make it cheaper.

This lesson covers CV0-004 objective 1.8: billing models (dedicated host, reserved resources, pay-as-you-go and spot), resource metering, tagging and rightsizing. It walks one pricing decision end to end with invented, round prices, because real prices change constantly; the lab sends you to the providers' own calculators for current ones.

Four ways to pay for compute

Two rows need a word more. Spot capacity is the provider's unused capacity, so its price is low and its availability is not promised; Microsoft states there is no SLA for Azure Spot VMs and that they are evicted with 30 seconds' notice when Azure needs the capacity back. It suits work that can stop and restart: batch processing, rendering, test environments. Dedicated hosts exist for two reasons that have little to do with price. AWS says they let you bring existing per-socket, per-core or per-VM software licenses, and that they help with compliance requirements; Google describes a sole-tenant node as a physical server dedicated to hosting only your project's VMs.

Procedure: price one workload three ways

Use invented round numbers, chosen only to make the arithmetic visible: on-demand costs $0.10 an hour, and a one-year commitment costs the equivalent of $0.065 an hour for every hour of the year, used or not.

1. Count the hours. A year has 8,760 hours. The workload, a web application, runs around the clock, so it uses all of them.
2. Price pay-as-you-go. 8,760 hours times $0.10 is $876.00.
3. Price the commitment. 8,760 times $0.065 is $569.40, paid whatever happens.
4. Find the break-even. At $0.10 an hour, on-demand reaches $569.40 after 5,694 hours, about 65 percent of the year. Any server that runs more than that saves money under the commitment; any that runs less loses money.
5. Decide. At 100 percent utilization the commitment saves $306.60 a year, about 35 percent. Commit.

The second run: change the hours, change the answer

Now the workload is an internal reporting server used only during working hours: eight hours a day on about 261 weekdays, 2,088 hours a year. Pay-as-you-go costs 2,088 times $0.10, which is $208.80. The commitment still costs $569.40, nearly three times as much, because it bills for 6,672 hours nobody used. The same discount that saved 35 percent now wastes money. Commitments are a bet on utilization, and the break-even point is the number to know.

Change one more input: a nightly batch job that renders thousands of independent frames and can restart any of them. At an invented spot price of $0.03 an hour, its 2,088 hours cost $62.64, provided the job checkpoints its work and tolerates interruptions. And if the reporting server's database needs a per-core license that only counts physical cores, the question stops being about the hourly rate at all: a dedicated host may be the only compliant way to use that license in the cloud.

Why this matters: the cheapest model depends on how many hours a workload actually runs and whether it can be interrupted. Commit only above break-even, use spot only for interruptible work, and use dedicated hosts for licensing or isolation, not price.

Metering: what the meter counts

Lesson 2 named measured service as one of NIST's five characteristics of the cloud. Resource metering is how it becomes a bill: every service counts something, and the bill multiplies each count by a rate. Compute is metered in time, storage in gigabyte-months, requests in thousands or millions, and data transfer in gigabytes moved. AWS's description of a stopped instance shows why knowing the units matters: you are not charged for usage or data transfer while an EC2 instance is stopped, but charges continue for its EBS storage, because the volume still exists. A developer who stops instances every evening has cut the compute meter, not the storage meter. When an instance runs, AWS bills at least one minute, then by the second.

Tagging: whose cost is this?

A bill that says the company spent a large sum on compute answers no useful question. Which product, which team, which environment? Tags, called labels in Google Cloud, answer it. They are key-value pairs attached to resources, such as CostCenter=4410, Environment=Production, Owner=payments-team or Application=checkout. Google states that label information is forwarded to the billing system so that charges can be broken down by label. AWS adds a step people miss: both AWS-generated and user-defined tags must be activated as cost allocation tags before they appear in Cost Explorer or in cost allocation reports. Microsoft's guidance adds two cautions that apply everywhere: use tag policies to make sure resources are created with the required tags, and never put sensitive values in tags, because they are stored as plain text.

Rightsizing: fitting the size to the load

Rightsizing means matching a resource's size to what it actually uses. Each provider offers a tool that reads utilization history and recommends changes: AWS Compute Optimizer analyzes resources' configuration and utilization metrics to recommend rightsizing and identify idle resources; Azure Advisor recommends shutting down or resizing underutilized virtual machines; Google Compute Engine offers machine type recommendations.

Work an example the way those tools do. A VM with 8 vCPUs averages 9 percent CPU over thirty days and peaks at 22 percent. Sizing to the average would be a mistake, because the peak matters: 22 percent of 8 vCPUs is about 1.8 vCPUs of real demand. On 1 vCPU that peak would saturate it; on 2 vCPUs it would run near 88 percent; on 4 vCPUs it sits near 44 percent, leaving headroom. Move to 4 vCPUs and the compute meter for this server roughly halves. Microsoft's Advisor, for instance, evaluates the 95th percentile of maximum CPU utilization rather than the average for exactly this reason.

Order matters. Rightsize first, then commit: a three-year commitment bought for an oversized fleet locks the waste in for three years.

Why this matters: tagging tells you whose cost it is, rightsizing removes what nobody uses, and commitments discount what remains, in that order.

Common misconceptions

• "Reserved capacity is always cheaper." Only above the break-even utilization; below it, a commitment costs more than paying as you go.
• "Spot is just cheap on-demand." It can be taken back with two minutes' warning on AWS and 30 seconds on Azure, with no SLA, so only interruptible work belongs there.
• "Stopping a VM stops all charges." Compute stops; attached storage keeps billing until it is deleted.
• "Average utilization is enough to rightsize." Size for peaks or a high percentile, or the smaller server fails at the busiest moment.

What you now know

• Pay-as-you-go bills by the second or hour; commitments discount steady use for one or three years; spot discounts interruptible use; dedicated hosts serve licensing and isolation.
• A commitment pays off only above its break-even utilization, about 65 percent of the year in this lesson's invented example.
• Metering multiplies counted units by rates; a stopped instance stops compute charges but not storage charges.
• Tags or labels allocate cost by team, product and environment; on AWS they must be activated for billing.
• Rightsize to peaks, not averages, and before buying commitments.

Sources

1. Amazon Web Services. (n.d.). Amazon EC2 On-Demand pricing and Amazon EC2 billing and purchasing options. aws.amazon.com; docs.aws.amazon.com
2. Amazon Web Services. (n.d.). Spot Instance interruption notices and How Amazon EC2 instance stop and start works. docs.aws.amazon.com (Spot); docs.aws.amazon.com (stop and start)
3. Microsoft. (n.d.). What are Azure Reservations?, Use Azure Spot Virtual Machines and Azure Dedicated Host. Microsoft Learn. learn.microsoft.com (reservations); learn.microsoft.com (Spot); learn.microsoft.com (Dedicated Host)
4. Google Cloud. (n.d.). Spot VMs, Committed use discounts and Sole-tenant nodes. docs.cloud.google.com (Spot); docs.cloud.google.com (CUDs); docs.cloud.google.com (sole-tenant)
5. Amazon Web Services, Google Cloud and Microsoft. (n.d.). Organizing and tracking costs using AWS cost allocation tags; Labels overview; Use tags to organize your Azure resources. docs.aws.amazon.com; docs.cloud.google.com; learn.microsoft.com
6. Amazon Web Services and Microsoft. (n.d.). What is AWS Compute Optimizer? and Optimize VM spend by resizing or shutting down underutilized instances (Azure Advisor). docs.aws.amazon.com; learn.microsoft.com
7. Amazon Web Services. (n.d.). What is AWS Pricing Calculator? docs.aws.amazon.com

A protocol from 1999

In 1999 Andy Stanford-Clark of IBM and Arlen Nipper of Arcom, now Eurotech, invented a messaging protocol for a narrow problem: getting small amounts of data from constrained devices across networks that were slow, high-latency or unreliable. They called it MQTT. Its design principles, as the project's own FAQ describes them, were to minimize network bandwidth and device resources while still giving some assurance of delivery. Those principles turned out to suit a world that did not yet exist. OASIS approved MQTT 3.1.1 as a standard on 29 October 2014, later ratified by ISO as well, and MQTT 5.0 on 7 March 2019. Today it is a protocol that both AWS IoT Core and Azure IoT Hub accept from devices.

The story of that protocol is a good thread to follow through the newest material on the exam, because the Internet of Things and cloud AI services meet in the same place: data from the physical world arriving in the cloud and being turned into decisions. This lesson follows one illustrative business, a fleet of refrigerated delivery trucks, from sensor to summary. It covers CV0-004 objective 1.11, identifying evolving technologies: machine learning and AI services, from text recognition to generative AI, and the IoT's sensors, gateways, communication and transmission protocols. Because this is the fastest-changing part of the cloud, expect product names to change; the capabilities are what the exam asks you to recognize.

The sensor

Start in the back of one truck. A sensor is a device that measures something physical, here temperature, and turns it into data: a reading of 4.2 degrees Celsius, stamped with a time and the sensor's identity. Sensors are often tiny and battery-powered, with little memory and processing power. That is what "constrained device" means, and it shapes every choice that follows. A sensor cannot run a full web browser's worth of software or afford to keep a radio on all day.

The communication link

The reading must leave the sensor over some short- or long-range radio. Inside the truck, Bluetooth Low Energy is enough to reach a box in the cab. On a farm, where soil sensors sit kilometres apart and must run for years, the LoRa Alliance describes LoRaWAN as a low-power, wide-area networking specification for battery-operated devices, with ranges of up to 15 km in rural areas. Wi-Fi suits devices near an access point and with mains power; cellular links suit things that move across a country, like the truck itself. The choice trades range, power use and data rate against each other.

The gateway

In the cab sits a gateway: a small computer that collects readings from all the truck's sensors and speaks to the cloud on their behalf. It earns its place in three ways. It aggregates, sending one average a minute instead of sixty raw readings and cutting the cellular bill. It filters, forwarding only readings that matter, such as a temperature rising above the safe limit. And it keeps working when the truck is in a tunnel, storing data until the connection returns. AWS describes its IoT Greengrass software as exactly this: an open-source edge runtime that lets devices act locally on the data they generate, run predictions from machine learning models, and filter and aggregate device data, operating autonomously when disconnected. It is the edge computing of Lesson 4, one truck at a time.

What matters here: sensors measure, links carry, and gateways aggregate, filter and buffer so constrained devices and costly connections do as little as possible.

The transmission protocol

From the gateway to the cloud, the reading travels in a protocol built for constrained links. MQTT is the most common: the OASIS standard calls it a client-server publish/subscribe messaging transport that is lightweight, open and simple, designed for constrained environments such as machine-to-machine communication. The gateway publishes to a topic such as fleet/truck7/temp; any service subscribed to that topic receives it, the fan-out pattern from Lesson 11. MQTT also lets a publisher ask for delivery assurance with a quality-of-service level, and mark a message as retained, so a new subscriber immediately receives the last known value.

Two other protocols appear in exam scenarios. CoAP, defined in RFC 7252, is a specialized web transfer protocol for constrained nodes, such as 8-bit microcontrollers with limited RAM and ROM, and constrained networks; it runs over UDP and borrows HTTP's request-and-response style. And plain HTTPS works for devices powerful enough to afford it, though Microsoft notes that because HTTPS has no efficient way to push messages to a device, devices using it must poll for cloud-to-device messages. On the cloud side, AWS IoT Core accepts MQTT, MQTT over secure WebSockets and HTTPS, and Azure IoT Hub accepts MQTT, AMQP, both over WebSockets as well, and HTTPS.

When the data arrives: AI services in the delivery business

The readings land in a cloud IoT service and flow on to storage and analytics. The same business now meets the other half of the objective: prebuilt AI services, called through an API and billed per request, with no model to train. Follow one delivery.

• Text recognition. The driver photographs a paper delivery note. Optical character recognition extracts its printed and handwritten text and the structure of its fields.
• Text translation. The customer's handwritten note is in Portuguese; a translation service turns it into English for the office.
• Visual recognition. A photo of damaged packaging is labelled automatically, so claims with damage are flagged.
• Sentiment analysis. Customer emails are scored for tone. Amazon Comprehend, for example, labels text positive, negative, neutral or mixed and returns a confidence score for each, so negative messages can reach a supervisor first.
• Voice-to-text. Drivers dictate end-of-shift reports, which become searchable text.
• Text-to-voice. The gateway's alert, "rear door open on truck seven", is spoken aloud in the cab.
• Generative AI. On Friday a manager asks for a summary of the week's temperature incidents in plain language, and a foundation model writes it from the stored records.

The last row shows how fast names move. AWS describes Bedrock as a fully managed service giving access to foundation models from many companies. Microsoft's Foundry, which Azure OpenAI resources can be upgraded into, unifies agents, models and tools. Google's machine learning platform documentation, long published as Vertex AI, now redirects to Gemini Enterprise Agent Platform. On the exam, recognize the capability; in practice, check the provider's current page.

The risks that come with the convenience

NIST's Generative AI Profile, AI 600-1 from July 2024, names the risks a cloud team should plan for. The best known is confabulation, which NIST defines as confidently stated but erroneous or false content, colloquially called hallucinations: the Friday summary might cite an incident that never happened, so generated output that drives decisions needs review against the source records. Data privacy is another: prebuilt AI services process your data on the provider's systems, so a hospital sending patient voice notes to a speech service must know where the data is processed and stored and whether that meets its obligations, the subject of Lesson 26. NIST also lists information security and intellectual property among the risks, both of which apply to any service that receives your documents.

What matters here: prebuilt AI services buy capability without building models, at the price of sending data to the provider and checking what comes back.

Common misconceptions

• "IoT devices should send every reading straight to the cloud." Gateways aggregate, filter and buffer so constrained devices and costly links carry only what matters.
• "MQTT is just HTTP for devices." It is publish/subscribe over a persistent connection, built for constrained devices and unreliable networks; CoAP is the HTTP-like option for very small devices over UDP.
• "Using an AI API means training a model." Prebuilt services such as OCR, translation and sentiment analysis are called and billed per request with no training.
• "Generative AI output is a reliable record." NIST's confabulation risk means it can be confidently wrong; verify before acting.

The short version

• IoT data flows from sensors over links such as Bluetooth LE, Wi-Fi, LoRaWAN or cellular to gateways that aggregate, filter and buffer it, then to the cloud.
• MQTT (1999; OASIS standards 2014 and 2019) is the lightweight publish/subscribe protocol for constrained devices; CoAP serves tiny devices over UDP; HTTPS suits capable ones.
• Prebuilt AI services cover text recognition, translation, visual recognition, sentiment analysis, voice-to-text and text-to-voice; Bedrock, Foundry and Google's Agent Platform host generative models.
• Plan for confabulation and data privacy before AI output or sensitive data leaves your control.

Sources

1. MQTT.org. (n.d.). Frequently asked questions. mqtt.org
2. Banks, A., Briggs, E., Borgendale, K., and Gupta, R. (Eds.). (2019). MQTT version 5.0 (OASIS Standard, 7 March 2019). OASIS. docs.oasis-open.org
3. Shelby, Z., Hartke, K., and Bormann, C. (2014). The Constrained Application Protocol (CoAP) (RFC 7252). RFC Editor. rfc-editor.org
4. LoRa Alliance. (n.d.). About LoRaWAN. lora-alliance.org
5. Amazon Web Services. (n.d.). Device communication protocols (AWS IoT Core) and What is AWS IoT Greengrass? docs.aws.amazon.com (IoT Core); docs.aws.amazon.com (Greengrass)
6. Microsoft. (n.d.). Choose a device communication protocol (Azure IoT Hub) and What is Microsoft Foundry? Microsoft Learn. learn.microsoft.com (IoT Hub); learn.microsoft.com (Foundry)
7. Amazon Web Services. (n.d.). Sentiment (Amazon Comprehend) and What is Amazon Bedrock? docs.aws.amazon.com (Comprehend); docs.aws.amazon.com (Bedrock)
8. Google Cloud. (n.d.). Compare AWS and Azure services to Google Cloud (AI and machine learning) and Introduction to machine learning on Gemini Enterprise Agent Platform. docs.cloud.google.com (comparison); docs.cloud.google.com (Agent Platform)
9. National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). doi.org

One question, two providers, opposite emphasis

AWS's prescriptive guidance for large migrations contains a blunt paragraph. Refactoring, it says, is not recommended for large migrations, because it means modernizing each application during the move, the most complex strategy and hard to manage across many applications. Instead, AWS recommends rehosting, relocating or replatforming first and modernizing after the migration is complete, since applications are easier to optimize or re-architect once they are already running in the cloud.

Microsoft's Cloud Adoption Framework, describing the same choice, warns in the other direction. Rehosting enables a fast, low-risk migration, it says, but don't rehost problematic workloads: rehosting doesn't resolve existing performance, reliability or architectural issues. Moving such workloads without modernization carries technical debt forward and requires rework later. It suggests rehosting only workloads you are confident will stay as they are for at least two years.

Both are right, about different applications. This lesson lays out the two positions and the evidence for each, then the directions a migration can take and the considerations that decide each application's fate. It covers CV0-004 objective 2.3: migration types, resource allocation, migration considerations and application migration strategies.

The strategies, side by side

CompTIA lists six strategies. The providers use more, and slightly different, names.

Two more names appear in provider guidance and on real migration plans. Repurchase, which Microsoft calls replace, swaps an application for a different product, usually SaaS; AWS nicknames it drop and shop. Relocate, in AWS's list, moves many servers at once from an on-premises platform to its cloud version, or between VPCs, regions or accounts.

The case for moving first

AWS's argument rests on risk and time. A migration with a fixed deadline, such as a data center lease ending, cannot afford to redesign hundreds of applications at once. Rehosting is predictable: tools such as AWS Application Migration Service perform continuous block-level replication of source servers and convert them to run in the cloud, with cutover windows typically measured in minutes. Once applications run in the cloud, the team learns the platform, managed services are one step away, and modernization proceeds application by application without a deadline overhead.

The case for modernizing on the way

Microsoft's argument rests on what rehosting carries with it. An application that crashes weekly from a memory leak will crash weekly in the cloud too, now with a monthly bill. Rehosted servers are often sized like the hardware they left and run around the clock, so a rehost can cost more than the data center did, which is why Lesson 14 put rightsizing before commitments. And "we will modernize later" often becomes never, because a working, rehosted application rarely wins priority over new features. Microsoft's two-year test is the practical rule: if the workload will need modernization within two years anyway, do it during the move, once.

Remember: the strategy is chosen per application, not per migration. Rehost the healthy and stable, replatform where a managed service removes real work, modernize the problematic and strategic, retain what cannot yet move, and retire what nobody uses.

Three directions of travel

The second row is not hypothetical. On 19 October 2022 David Heinemeier Hansson of 37signals, the company behind Basecamp and HEY, announced that it was leaving the cloud, concluding that renting computers is mostly a bad deal for medium-sized companies with stable growth. He agreed that the cloud is excellent when load is highly irregular, citing HEY's launch, when 300,000 users signed up in three weeks against a forecast of 30,000 in six months. That is the break-even logic of Lesson 14 applied to a whole company: elasticity is worth paying for when you use it.

Leaving any provider used to carry a toll, the charge for moving data out. On 5 March 2024 AWS announced it would waive data-transfer-out charges for customers moving their data off AWS, following the direction set by the European Data Act. It noted that most customers already pay nothing for their first 100 GB a month. Lower exit costs reduce one part of vendor lock-in; the larger part, dependence on services with no equivalent elsewhere, remains a design decision.

Considerations, application by application

CompTIA's list of considerations works as a checklist for each application.

• Platform compatibility. Will the operating system, CPU architecture and licenses run on the target? Disk formats matter too: AWS VM Import/Export accepts OVA, stream-optimized VMDK, fixed and dynamic VHD and VHDX, and raw images, while Azure requires uploaded disks in VHD format with a fixed size.
• Compute and storage, and resource allocation. Size targets from measured utilization, not from the old server's specification, and map each disk to the right block, file or object service and tier (Lesson 12).
• Networking. Address ranges must not overlap (Lesson 5), connectivity must exist before cutover (Lesson 6), and bulk data transfer needs bandwidth and time.
• Cost and management overhead. Compare the full run cost, including people, not just instance prices; managed services lower overhead, rehosted VMs keep it.
• Service availability. Plan the cutover window and rollback, and use replication where downtime must be short.
• Vendor lock-in. Note every proprietary service the design depends on and what leaving would require.
• Environmental: power and cooling. Data centers spend energy beyond the computers themselves. Power usage effectiveness, PUE, divides a facility's total energy by the energy reaching IT equipment, so 1.0 would mean no overhead; Google reports a 2025 fleet average of 1.09.
• Regulatory and compliance. Data residency and industry rules can force retain, or a particular region, as Lessons 3 and 26 discuss.

Common misconceptions

• "Lift and shift is always the wrong approach." AWS recommends it for most applications in large migrations; the error is rehosting a problematic workload that needs modernizing anyway.
• "Moving to the cloud fixes an application's problems." Microsoft's warning is explicit: rehosting doesn't resolve performance, reliability or architectural issues.
• "Migration only goes one way." Cloud-to-on-premises and cloud-to-cloud moves are real, as 37signals' exit shows.
• "Size cloud servers to match the old hardware." Old servers are often oversized; measured utilization, and retiring zombie applications, should drive allocation.

Putting it together

• Rehost moves as is; replatform changes the hosting with minimal code change; refactor changes code; re-architect changes architecture and code; retain waits; retire switches off.
• AWS favours moving first and modernizing after for large migrations; Microsoft warns against rehosting problematic workloads. Decide per application, using health, deadline and a two-year modernization horizon.
• Migrations run on premises to cloud, cloud to on premises and cloud to cloud; exit data charges have fallen, proprietary dependencies have not.
• Check compatibility, sizing, networking, cost, overhead, availability, lock-in, power and cooling, regulation and compliance for every application.

Sources

1. Amazon Web Services. (n.d.). Migration strategies. AWS Prescriptive Guidance, Guide for AWS large migrations. docs.aws.amazon.com
2. Microsoft. (n.d.). Select your cloud migration strategies. Cloud Adoption Framework for Azure. learn.microsoft.com
3. Amazon Web Services. (n.d.). What is AWS Application Migration Service? docs.aws.amazon.com
4. Heinemeier Hansson, D. (2022, October 19). Why we're leaving the cloud. HEY World. world.hey.com
5. Stormacq, S. (2024, March 5). Free data transfer out to internet when moving out of AWS. AWS News Blog. aws.amazon.com
6. Amazon Web Services and Microsoft. (n.d.). Image formats supported by VM Import/Export; Prepare a Windows VHD or VHDX to upload to Azure. docs.aws.amazon.com; learn.microsoft.com
7. Google. (n.d.). Power usage effectiveness. Google Data Centers. datacenters.google
8. Oracle. (n.d.). VBoxManage export and VBoxManage clonemedium. Oracle VirtualBox user guide. virtualbox.org

A service that reached its end

On 31 March 2026 Google Cloud Deployment Manager, Google's original service for defining infrastructure in template files, reached end of support, and Google told its users to migrate to Infrastructure Manager or another deployment technology. For teams whose infrastructure lived in templates, that meant a translation job: the files said exactly what existed, and they could be rewritten. For a team whose infrastructure existed only as settings someone had clicked into a console over the years, there would have been nothing to translate, only an inventory to reconstruct by hand.

That difference is the case for infrastructure as code, IaC: describing infrastructure in files that a tool reads and makes real. This lesson walks one small deployment end to end with OpenTofu, then changes something behind the tool's back to show drift. It covers the infrastructure-as-code half of CV0-004 objective 2.4: IaC, repeatability, drift detection, versioning, testing and documentation, along with Terraform, one of the DevOps tools in objective 5.4. Lesson 18 covers the other half, configuration as code and scripting.

What infrastructure as code is

HashiCorp describes Terraform as an infrastructure as code tool that lets you build, change and version cloud and on-premises resources safely and efficiently, defining them in human-readable configuration files that you can version, reuse and share, and managing them with a consistent workflow. OpenTofu's documentation uses nearly identical words, because OpenTofu is a fork of Terraform. Its FAQ explains that companies including Gruntwork, Spacelift, Harness, env0 and Scalr started it in response to HashiCorp's switch of Terraform from an open-source license to the Business Source License. The two share the workflow and the configuration language this lesson uses, and either works for the lab.

The workflow has three steps. You write configuration declaring the resources you want. You plan: the tool compares your configuration with what exists and produces an execution plan listing what it will create, update or destroy. You apply the plan. The approach is declarative: you describe the end state, not the steps to reach it, and the tool works out the steps. To do that it keeps state. Terraform's documentation says the tool uses state to map real-world resources to your configuration, keep track of metadata and improve performance, and that state is how it determines which changes to make.

Procedure: declare, plan, apply

To keep the lab free and local, the resources are Docker containers on your own machine, managed through the kreuzwerker Docker provider. In a cloud, the same file would name VPCs and instances instead; nothing else about the workflow changes. Save this as main.tf in an empty folder:

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 4.6"
    }
  }
}

provider "docker" {}

variable "web_port" {
  description = "Host port that publishes the web container"
  type        = number
  default     = 8085
}

resource "docker_image" "web" {
  name = "nginx:alpine"
}

resource "docker_container" "web" {
  name  = "iac-web"
  image = docker_image.web.image_id
  ports {
    internal = 80
    external = var.web_port
  }
}

1. Initialize. tofu init downloads the provider and writes a dependency lock file, .terraform.lock.hcl, recording the exact provider version chosen.
2. Check. tofu fmt normalizes formatting and tofu validate checks the configuration is internally consistent, before anything touches real resources.
3. Plan. tofu plan reports that it will add two resources, the image and the container, and shows every attribute it will set. This is the review step: read it.
4. Apply. tofu apply shows the plan again, asks for confirmation and creates both resources. curl -s localhost:8085 now returns the nginx welcome page, and tofu state list shows the two resources the tool is tracking.
5. Apply again. Run tofu plan once more. It reports no changes. The configuration describes an end state that already exists, so there is nothing to do: running the same code twice gives the same result. That is repeatability, and it is what makes IaC safe to run often.

The second run: drift

Now change one input, from outside the tool. Someone removes the container by hand with docker rm -f iac-web. The configuration still says the container should exist, but reality says it does not. That gap is drift.

The next tofu plan catches it. Planning refreshes the state against the real objects, finds the container missing and proposes to create it again. If you only want to see what changed outside the tool, without proposing any fixes, OpenTofu's refresh-only mode, tofu plan -refresh-only, creates a plan whose only goal is to update the state to match changes made to remote objects outside OpenTofu. Apply the normal plan and the container is back: the code, not anyone's memory, decides what should exist.

Drift in real clouds is usually subtler: a firewall rule opened during an incident and never closed, an instance resized in the console. Teams detect it by running plans on a schedule, often in the pipelines of Lesson 31, and treating any unexpected difference as an alert. Then they either revert the manual change by applying the code, or bring the change into the code so the two agree again.

In short: plan compares declared state with real state. Drift is any difference you did not make through the code, and a scheduled plan is the detector.

Repeatability, versioning, testing and documentation

• Repeatability. Variables let one configuration produce several environments. tofu apply -var web_port=8086 builds the same stack on a different port; in a cloud, the same variables would choose instance sizes for dev, test and production. Because changing a container's published port cannot be done in place, the plan shows the container being replaced, a reminder to read plans before applying them.
• Versioning. Commit the configuration and the .terraform.lock.hcl file to Git. OpenTofu's documentation describes the lock file as belonging with the configuration, so every run uses the same provider versions; the version constraint in main.tf allows updates only within the 4.x series. Keep state itself out of Git: teams store it in a shared remote backend with locking, so two people applying at once cannot overwrite each other.
• Testing. Beyond fmt and validate, tofu test runs test files containing run blocks and assertions. OpenTofu's documentation says it creates real infrastructure, checks that the required conditions hold, and destroys what it created when the test completes.
• Documentation. The configuration is itself documentation of what exists, which is the point of the opening example. Descriptions on variables, comments explaining why a setting has its value, and a short README that says how to run the code complete it.

Common misconceptions

• "Infrastructure as code is a script of console clicks." Declarative IaC states the end result; the tool computes the steps and does nothing when the result already exists.
• "Once applied, infrastructure stays as declared." Manual changes create drift, which only a fresh plan reveals.
• "The state file is an optional cache." State is how the tool maps configuration to real resources and decides what to change; losing or splitting it causes real damage.
• "Plan is a formality before apply." The plan is the review: it can show a replacement or a deletion you did not intend.

Where this leaves us

• IaC declares infrastructure in versioned files and applies it through write, plan and apply; Terraform and OpenTofu work across providers, while CloudFormation, Bicep and Infrastructure Manager are provider-specific.
• State maps real resources to configuration; planning twice with no changes shows repeatability.
• Drift is change made outside the code; plans, including refresh-only plans, detect it, and scheduled plans catch it early.
• Version the code and its lock file, keep state in a locked remote backend, test with validate and tofu test, and document with descriptions and comments.

Sources

1. HashiCorp. (n.d.). What is Terraform? and State. Terraform documentation. developer.hashicorp.com (intro); developer.hashicorp.com (state)
2. OpenTofu. (n.d.). What is OpenTofu? and Frequently asked questions. opentofu.org (intro); opentofu.org (FAQ)
3. OpenTofu. (n.d.). Command: plan, Command: test and Dependency lock file. opentofu.org (plan); opentofu.org (test); opentofu.org (lock file)
4. kreuzwerker. (n.d.). Terraform provider for Docker [README and examples]. github.com
5. Amazon Web Services. (n.d.). What is CloudFormation? docs.aws.amazon.com
6. Microsoft. (n.d.). What is Bicep? Microsoft Learn. learn.microsoft.com
7. Google Cloud. (n.d.). Cloud Deployment Manager documentation (end-of-support notice). docs.cloud.google.com

A script that worked once

Here is a plausible wrong answer, heard after many outages: "The setup script ran perfectly on the first server yesterday, so running it again on this one is safe." The script contains a line like echo "APP_MODE=prod" >> /etc/app.conf, which appends a setting to a configuration file. Run it once and the file is right. Run it again, on a server that was half set up or by a colleague who did not know it had already run, and the file now holds the line twice. Run it a third time and a step that creates a user fails, because the user already exists, and the script stops halfway.

Trace where the reasoning failed. The script describes steps, not a result, so each run repeats the steps whatever the state of the machine. Ansible's glossary names the property the script lacks: an operation is idempotent if the result of performing it once is exactly the same as the result of performing it repeatedly. This lesson follows that wrong answer and two others, about data formats and about comparisons, through to their fixes. It covers the configuration-as-code half of CV0-004 objective 2.4: configuration as code, scripting logic with variables, conditionals, operators, data types and functions, and the JSON and YAML formats, together with Ansible from objective 5.4.

Configuration as code: the layer after infrastructure

Lesson 17's infrastructure as code creates the servers, networks and storage. Configuration as code, CaC, decides what is on and inside them: packages, files, settings, users and services, declared in versioned files and applied by a tool. Ansible is the tool in the exam's list. Its documentation describes playbooks as simple, human-readable scripts written in YAML, and an agentless design: Ansible uses SSH with existing operating system credentials to reach the machines it manages, so there is no extra software to install on them. Its control node needs Linux, macOS or WSL, as Lesson 1 noted.

The heart of the difference from a shell script is the module. A task does not say "append this line"; it says "this line must be present", and the module checks before acting. The lineinfile module, in its own documentation's words, ensures a particular line is in a file. Here is the fix for the opening script as a playbook:

- name: Configure the application host
  hosts: localhost
  connection: local
  vars:
    app_mode: prod
  tasks:
    - name: Ensure the configuration directory exists
      ansible.builtin.file:
        path: /tmp/cac-lab
        state: directory
    - name: Ensure the mode setting is present exactly once
      ansible.builtin.lineinfile:
        path: /tmp/cac-lab/app.conf
        line: "APP_MODE={{ app_mode }}"
        create: true

The first run reports two tasks changed. Every later run reports them as ok with zero changed, because both checks find the declared state already true. Change the variable to staging and only the one affected setting changes. Two more options make playbooks safer to run. Check mode, --check, is what Ansible's glossary calls a dry run: it reports what would change without changing anything, and --diff shows the lines involved. And --syntax-check parses the playbook without running it, catching format errors before they reach a server.

The upshot: configuration as code replaces "do these steps" with "make this true", so running it again is safe and running it everywhere gives the same result.

The second wrong answer: "the YAML looks fine"

A playbook that looks perfectly aligned fails to parse. The cause is usually invisible: a tab. The YAML 1.2.2 specification says that to maintain portability, tab characters must not be used in indentation. Indentation is not decoration in YAML; it is structure, so a task indented two spaces too far belongs to a different parent than you intended, and a mixed tab breaks the file outright.

The other YAML trap is typing. YAML guesses each unquoted value's type. Under the core schema an unquoted 1.10 matches the pattern for a floating-point number, so a version field written as version: 1.10 is read as the number 1.1, and the release you asked for silently becomes a different one. Quoting the value, version: "1.10", makes it a string. The rule of thumb is to quote anything that looks like a number but is really a label, such as versions, postal codes and account IDs.

JSON, the other format in the objective, is stricter and simpler. RFC 8259 defines four primitive types, strings, numbers, booleans and null, and two structured types, objects and arrays. Its grammar has no comments and requires double quotes around strings and object keys. Cloud APIs, CloudFormation templates and IAM policies use JSON heavily; YAML's specification says version 1.2 was designed to be a strict superset of JSON, which is why many YAML tools also read JSON.

The third wrong answer: "the comparison worked in testing"

A Bash script decides whether to scale out by comparing a queue length with a threshold of 9. In testing, with queue lengths of 3 and 7, it behaved. In production the queue reached 10, and the script decided 10 was smaller than 9. The line read if [[ "$queue" < "9" ]].

The bash manual explains the failure. Inside double brackets, the < operator compares strings, which it does lexicographically using the current locale, character by character, and the character 1 sorts before 9. Numbers need arithmetic operators: in a conditional expression the manual lists -eq, -ne, -lt, -le, -gt and -ge as the arithmetic comparisons, and if [[ "$queue" -lt 9 ]], or the arithmetic form (( queue < 9 )), gives the right answer. The general lesson is about data types. Bash stores variables as strings and treats them as numbers only in arithmetic contexts, while JSON and YAML values carry types of their own, so a script must know which kind of value it holds.

The same script shows the other pieces of scripting logic the objective names.

• Variables are assigned with no spaces around the equals sign, as in queue=10, and read as "$queue". The quotes matter: the manual explains that unquoted expansions undergo word splitting, so a file name such as Q3 report.txt, unquoted, becomes two arguments.
• Conditionals such as if, then, else and fi choose between actions based on a test's result.
• Operators include comparisons, arithmetic and logical and and or; choosing string or numeric operators is a data-type decision.
• Functions name a block of commands so it can be reused with arguments, which read as $1, $2 and so on inside the function. A function such as ensure_line, which checks with grep -q whether a line is present and appends it only if not, makes the opening script idempotent by hand.

The upshot: most scripting bugs are type and quoting bugs. Compare numbers as numbers, quote every variable, and check before you change.

Versioning, testing and documenting configuration code

Configuration code earns the same habits as infrastructure code. Keep playbooks and scripts in Git and change them through review, as Lesson 31 describes. Test before production with --syntax-check and --check, and run a playbook twice in a test environment: if the second run reports anything changed, a task is not idempotent. Give every task a name that says what it ensures, because Ansible prints those names as it runs, so they double as documentation and as a log of what happened.

Common misconceptions

• "If a script ran once successfully, it is safe to run again." Only idempotent code is. Appending, creating and incrementing steps repeat their effect.
• "YAML indentation is cosmetic, and tabs are fine." Indentation is structure, and the specification forbids tabs in it.
• "JSON allows comments and unquoted keys like YAML." RFC 8259's grammar has neither; strings and keys need double quotes.
• "In Bash, < and -lt mean the same thing." Inside double brackets, < compares strings lexicographically; -lt compares numbers.

Pulling it together

• Configuration as code declares packages, files, settings and services; Ansible applies YAML playbooks over SSH without agents, and its modules are idempotent.
• Check mode previews changes; a second run reporting zero changed proves idempotency.
• YAML forbids tabs in indentation and guesses types, so quote labels that look like numbers; JSON has strict quoting, typed values and no comments.
• Scripting logic is variables, conditionals, operators, data types and functions; quote variables and use numeric operators for numbers.

Sources

1. Ansible project. (n.d.). Introduction to Ansible and Glossary (idempotency, check mode). Ansible documentation. docs.ansible.com (introduction); docs.ansible.com (glossary)
2. Ansible project. (n.d.). ansible-playbook and ansible.builtin.lineinfile module. Ansible documentation. docs.ansible.com (CLI); docs.ansible.com (lineinfile)
3. Ben-Kiki, O., Evans, C., and döt Net, I. (2021). YAML Ain't Markup Language (YAML) version 1.2.2. YAML Language Development Team. yaml.org
4. Bray, T. (Ed.). (2017). The JavaScript Object Notation (JSON) data interchange format (RFC 8259). RFC Editor. rfc-editor.org
5. Fox, B., and Ramey, C. (n.d.). bash(1): quoting, conditional expressions and functions [Manual page]. The Linux man-pages project. man7.org
6. jq project. (n.d.). jq manual. jqlang.org

Eight requirements and a deadline

The requirements sheet in front of you has a number in nearly every line. A learning platform must serve 300 requests a second most of the year and 2,000 a second during the two-week exam period. Its PostgreSQL database holds 500 GB and needs 10,000 IOPS at peak, mostly 8 KiB random reads and writes. It streams 20 TB of course video to as many as 1,000 students at once at about 5 Mbps each. Its availability target is 99.9 percent, with an RPO of 15 minutes and an RTO of one hour. Student records must be encrypted, must never be reachable from the internet, and by law must stay in the country. And there is a budget.

How do you turn that into resources? This lesson works the sheet line by line and then generalizes. It covers CV0-004 objective 2.5, provisioning cloud resources to a set of storage, performance, security, cost, availability, compliance, network and compute requirements. It also covers objective 1.10, comparing ways to optimize workloads with compute choices, storage IOPS and throughput, network latency and throughput, orchestration, workflow and managed services. Almost every earlier lesson contributes a piece.

Step 1: apply the constraints that cannot bend

Start with requirements that remove options rather than rank them. The residency rule means every resource holding student data runs in a region inside the country, which may also be the right answer for latency, since the students are there too. Security rules out a database with a public address: it goes in a data subnet with only a local route, reachable from the application tier alone (Lesson 5), with encryption at rest and TLS in transit (Lesson 28). Only after these constraints is it worth comparing prices, because the cheapest region in the world is irrelevant if the law forbids it.

Step 2: availability sets the shape

A 99.9 percent target allows about 43.8 minutes of downtime a month (Lesson 4), less than one bad zone failure would cost if everything sat in one zone. So the web tier runs in two zones behind a load balancer, and the database uses a managed Multi-AZ deployment with a standby in a second zone (Lesson 13). The 15-minute RPO is met by continuous backups with point-in-time recovery, and the one-hour RTO by automated failover within the region. A region-wide outage would break the target; covering it would need the costlier multi-region designs of Lesson 4, which this budget does not justify, and that trade-off should be written down rather than assumed.

Step 3: choose compute by the shape of each job

One platform has several kinds of work, and each suits a different form of compute.

The web and API tier, steady with a seasonal peak, runs as containers under an orchestrator (Lesson 10) or as VMs in an autoscaling group (Lesson 22). Thumbnail generation for uploaded photos, a few hundred times a day, is a textbook function: nothing runs or bills between uploads. The nightly analytics job can restart if interrupted, which makes it a candidate for spot capacity (Lesson 14). Orchestration is itself an optimization: packing many containers onto fewer VMs and replacing failed ones raises utilization without anyone watching.

Processes with several steps need coordination as well. An upload must be scanned, resized, recorded and announced, with retries when a step fails. Workflow services do this. AWS describes Step Functions workflows as a series of event-driven steps, and Azure Logic Apps orchestrates business process workflows across systems. Google's Workflows is a fully managed orchestration platform that runs steps in order or in parallel.

Step 4: size storage by IOPS and throughput, not just capacity

The database line hides a calculation. Throughput equals IOPS times I/O size, so 10,000 IOPS of 8 KiB operations move about 80,000 KiB per second, roughly 78 MiB/s. The hard part of this workload is the number of operations, not the volume of data. On AWS's gp3 volumes, the documentation says performance can be provisioned independently of volume size, from a baseline of 3,000 IOPS and 125 MiB/s, so you provision the extra IOPS on a 500 GB volume. The older gp2 type tied IOPS to size at 3 IOPS per GiB, which is why teams once bought far more space than they needed just to get speed: an optimization trap worth recognizing.

The video line is the opposite shape. Twenty terabytes read sequentially is a throughput workload, and 1,000 viewers at 5 Mbps is about 5 Gbps leaving the platform. Object storage holds the files cheaply, colder tiers can hold courses nobody has opened in a year (Lesson 12), and a CDN serves cached copies from edge locations so the origin carries only a fraction of the traffic (Lesson 7).

Step 5: network latency and network throughput

The two network numbers in the objective are easy to confuse. Latency is how long one round trip takes; throughput is how much data moves per second. They fail differently. Suppose a page makes 20 database queries one after another, and the database sits in another region 40 milliseconds away. The page waits about 800 milliseconds whatever the bandwidth, because each query must finish before the next begins. Buying a bigger link changes nothing; placing the application and database close together, or batching the queries, does. A video stream has the opposite problem: once it starts, a few extra milliseconds of latency hardly matter, but too little throughput stalls it.

Worth holding on to: for storage, multiply IOPS by I/O size to find throughput and size for whichever binds; for networks, count round trips to see whether latency or throughput is the real limit.

Step 6: cost, and the managed-service lever

With the shape fixed, the cost requirement is met with Lesson 14's tools. Commit to the steady baseline of the web tier, scale out on demand for the two-week peak, and run the nightly batch on spot capacity. Rightsize from measurements after launch, and tag everything by course and environment so costs can be traced. The largest saving is often staff time: the managed Multi-AZ database replaces patching, backups and failover work that a self-managed database would demand every week, which is why managed services appear in the objective as a workload optimization in their own right.

The requirements, resolved

Common misconceptions

• "Provisioning starts with the cheapest option." Constraints such as residency and security come first; price ranks only the options that remain.
• "More storage capacity means more speed." On volume types that provision performance separately, such as gp3, IOPS and throughput are bought independently of size.
• "A faster network link fixes slow, chatty pages." Sequential round trips are bound by latency, not bandwidth; move components closer or make fewer calls.
• "Serverless is always cheapest." Functions suit short, spiky work; steady, heavy load is usually cheaper on committed VMs or containers.

Summing up

• Work requirements in order: hard constraints (compliance, security), then availability, then compute, storage and network shape, then cost.
• Match compute to the job: VMs for control, containers for many replicas and frequent releases, functions for short event-driven work, spot for interruptible batch.
• Throughput equals IOPS times I/O size; provision whichever binds. Latency is per round trip, throughput per second, and they need different fixes.
• Orchestration packs and heals containers, workflow services coordinate multi-step processes, and managed services cut the operational work that often dominates cost.

Sources

1. Amazon Web Services. (n.d.). Amazon EBS General Purpose SSD volumes. Amazon EBS User Guide. docs.aws.amazon.com
2. Amazon Web Services. (n.d.). What is Step Functions? AWS Step Functions Developer Guide. docs.aws.amazon.com
3. Microsoft. (n.d.). What is Azure Logic Apps? Microsoft Learn. learn.microsoft.com
4. Google Cloud. (n.d.). Workflows overview. docs.cloud.google.com
5. Amazon Web Services. (n.d.). What is AWS Lambda? and Lambda quotas. docs.aws.amazon.com (overview); docs.aws.amazon.com (quotas)
6. Axboe, J., and contributors. (n.d.). fio: Flexible I/O tester documentation. fio.readthedocs.io
7. ESnet. (n.d.). iperf3 [Software and documentation]. github.com

Two articles and a default

On 1 March 2010 Martin Fowler described a technique he called blue-green deployment: keep two production environments, as identical as possible, with one of them, say blue, live. Deploy the new release to green and test it there; when it works, switch the router so all requests go to green and blue sits idle. If anything goes wrong, switch the router back. On 25 June 2014 Danilo Sato described another, the canary release: roll a change out to a small subset of users first. It is named after the canary that miners carried, which died of leaking gas before the miners did. And in every Kubernetes cluster today a third technique runs by default: a Deployment replaces Pods a few at a time, keeping at least 75 percent of them available.

Those are three of the four strategies in CV0-004 objective 2.2, which asks you to implement the appropriate deployment strategy for a scenario: blue-green, canary, rolling and in-place. Each answers the same question differently: how does a new version reach users, and what happens if it is bad?

The four strategies side by side

Reading the rows: in-place and rolling

An in-place deployment updates the servers that are already running. Done all at once, it is the simplest strategy and the most dangerous: every server stops, updates and restarts together, so users see an outage and every one of them meets a bad release. Kubernetes calls this the Recreate strategy, and its documentation is plain: all existing Pods are killed before new ones are created. AWS CodeDeploy offers in-place deployments at three speeds, all at once, half at a time and one at a time, trading speed for the capacity kept in service.

A rolling deployment is an in-place update done in careful batches, and it is Kubernetes' default. With default settings of 25 percent maximum unavailable and 25 percent maximum surge, a Deployment of 8 replicas keeps at least 6 Pods available during a rollout. It runs at most 10, creating new Pods before removing old ones. If the new version's Pods fail to become ready, the rollout stalls rather than taking everything down, and kubectl rollout undo returns to the previous version.

Rolling has one consequence that scenario questions test. Kubernetes notes that rolling updates run multiple versions of an application at the same time. For a while, old and new code serve users side by side against the same database and the same APIs. A release that changes a database schema in a way the old version cannot read therefore breaks the old Pods mid-rollout. Such changes need an intermediate, backward-compatible step, or a strategy that switches all traffic at once.

That intermediate step has a name. Danilo Sato describes parallel change, also known as expand and contract, as a way to make a backward-incompatible change safely in three phases: expand, migrate and contract. Applied to renaming a database column, it works like this. First add the new column and have the new release write to both columns (expand). Next, roll out the release and copy the old data across while old and new Pods still run side by side (migrate). Only when no running code reads the old column does a later release remove it (contract). Every release in the sequence can run alongside its neighbour, so rolling updates and canaries stay safe, and each step can be rolled back on its own.

So what? in-place is cheap and abrupt; rolling avoids downtime by keeping most capacity serving, at the price of two versions running together.

Reading the rows: blue-green and canary

Blue-green removes the mixed-version window by switching everything at once, and it buys the fastest rollback of any strategy: the old environment is still there, idle, so going back is one routing change. The costs are capacity, a complete second environment during the switch, and data. Fowler notes that the two environments must deal with transactions made while green was live, which is why many teams keep a shared database and make schema changes backward-compatible. Azure App Service builds the pattern in as deployment slots: separate live apps with their own host names, where you validate and warm up a release before swapping it into production, and swap back if needed.

A canary limits who is exposed, rather than how quickly you can recover. A small share of traffic, often a few percent, goes to the new version while you watch error rates and latency; if they hold, the share widens until it reaches everyone. AWS CodeDeploy names configurations after the pattern, such as one that sends 10 percent of Amazon ECS traffic to the new version for 15 minutes before shifting the rest. Google Cloud Run lets you specify what percentage of traffic each revision receives. Sato's article adds a caution: canaries can be used for A/B testing, but the concerns should not be conflated. A canary looks for regressions quickly; an A/B test compares variants until the result is statistically significant.

Choosing, scenario by scenario

1. Can two versions run at once? If a change is not backward-compatible with the data or APIs, avoid rolling and canary, or add a compatible intermediate release first.
2. How fast must rollback be? Seconds points to blue-green; minutes allows rolling or canary.
3. How many users may meet a bad release? If the answer is very few, use a canary and watch it.
4. Can you afford duplicate capacity? Blue-green needs a full copy during the switch; rolling needs only a small surge; in-place needs none.
5. Can you afford downtime? Only in-place all at once, or Recreate, accepts an outage, sometimes legitimately, for an internal tool updated after hours.

So what? rollback speed favours blue-green, limited exposure favours canary, low cost favours rolling, and simplicity favours in-place; a schema change can veto the strategies that run two versions together.

Common misconceptions

• "Rolling updates never mix versions." They do by design: old and new Pods serve together until the rollout finishes.
• "Blue-green is free insurance." It needs a full second environment during the switch and careful handling of shared data.
• "A canary and an A/B test are the same thing." A canary looks for regressions; an A/B test compares variants for business significance.
• "In-place deployment always means downtime." One-at-a-time in-place updates keep most capacity serving; all at once is what causes the outage.

Looking back

• In-place updates existing servers, all at once (Recreate) or in steps; rolling replaces instances in batches, Kubernetes' default at 25 percent unavailable and 25 percent surge.
• Blue-green runs two environments and switches the router, giving instant rollback at the cost of duplicate capacity; App Service slots implement it.
• Canary sends a small share of traffic to the new version and widens it while metrics hold; CodeDeploy configurations and Cloud Run traffic percentages implement it.
• Rolling and canary run two versions at once, so data and API changes must be backward-compatible.

Sources

1. Fowler, M. (2010, March 1). BlueGreenDeployment. martinfowler.com. martinfowler.com
2. Sato, D. (2014, May 13). ParallelChange. martinfowler.com. martinfowler.com
3. Sato, D. (2014, June 25). CanaryRelease. martinfowler.com. martinfowler.com
4. The Kubernetes Authors. (n.d.). Deployments. Kubernetes documentation. kubernetes.io
5. Amazon Web Services. (n.d.). Working with deployment configurations in CodeDeploy. AWS CodeDeploy User Guide. docs.aws.amazon.com
6. Microsoft. (n.d.). Set up staging environments in Azure App Service. Microsoft Learn. learn.microsoft.com
7. Google Cloud. (n.d.). Rollbacks, gradual rollouts, and traffic migration. Cloud Run documentation. docs.cloud.google.com

One or two engineers out of twelve

Google's Site Reliability Engineering book reports that a Google SRE team of 10 to 12 people typically has one, or sometimes two, members whose main assignment is building and maintaining monitoring for their service. That is how much work it takes to know, at any moment, whether a system is healthy and, when it is not, why. The goal has a name: observability. The OpenTelemetry project defines it as the ability to understand a system from the outside, by asking questions about it without knowing its inner workings.

This lesson builds observability for one service step by step: decide what to watch, collect and keep logs, record metrics, trace requests, and alert people only when they need to act. It covers CV0-004 objective 3.1, configuring resources to achieve observability through logging (collection, aggregation and retention), tracing, monitoring with metrics, and alerting with triage and response. It also introduces two DevOps tools from objective 5.4, the ELK stack and Grafana.

Three signals, three questions

Step 1: decide what to watch

Start from the user, not the servers. The SRE book's four golden signals are the core of most dashboards.

• Latency: the time it takes to service a request, tracking failed requests separately, since a fast error can flatter the average and a slow error is worse than a fast one.
• Traffic: how much demand the system faces, such as HTTP requests per second.
• Errors: the rate of requests that fail, explicitly with an error code, implicitly with wrong content, or by policy, such as any response slower than a promised second.
• Saturation: how full the service is, emphasizing its most constrained resource, whether memory, I/O or CPU, because many systems degrade before they reach 100 percent.

Host measurements such as CPU and disk still matter, mostly as inputs to saturation. The service's users, though, experience latency and errors, and those are what alerts should usually watch.

Step 2: collect, aggregate and retain logs

NIST's guide to log management, Special Publication 800-92, describes log management as generating, transmitting, storing, analyzing and disposing of log data, and the cloud adds urgency to each step. Collection means every component writes logs somewhere they can be gathered, ideally as structured records, such as JSON with named fields, rather than free text. Aggregation means shipping them to one central system, because cloud instances and containers are disposable. A log left on a server vanishes when the server is replaced, and twelve services' logs are useless if they cannot be searched together. NIST adds a requirement that is easy to forget: each logging host's clock must be synchronized to a common time source, or events from different systems cannot be put in order.

The ELK stack is the classic open-source aggregation pipeline. Elastic describes the stack as Elasticsearch, Kibana, Beats and Logstash. Beats are lightweight shippers that send data from many machines. Logstash is a server-side pipeline that ingests data from many sources, transforms it, for example by parsing structure out of free text or removing sensitive fields, and sends it on. Elasticsearch stores and indexes it for search, and Kibana explores and visualizes it. Each provider runs a managed equivalent, such as CloudWatch Logs, Azure Monitor Logs and Cloud Logging.

Retention decides how long logs are kept, which is a cost decision and often a compliance one. Defaults deserve checking. AWS states that CloudWatch Logs stores log data indefinitely by default and that retention can be changed for each log group at any time. A team that never sets it pays to keep every line forever. Set retention to what policy and law require, and archive older logs to cheaper storage if they must be kept for years.

Step 3: record metrics and draw them

Prometheus, an open-source monitoring and alerting toolkit, shows how metrics systems work. It collects and stores metrics as time series, each sample timestamped and tagged with key-value labels such as service and instance. It gathers them with a pull model, scraping each target over HTTP at regular intervals. Every scrape also records a synthetic metric, up, which is 1 if the target was reachable and 0 if the scrape failed: the simplest health signal there is. Its query language, PromQL, turns raw counters into rates, such as requests per second over the last minute.

Grafana turns those numbers into dashboards. Its documentation describes the open-source edition as letting you query, visualize, alert on and explore metrics, logs and traces wherever they are stored, through data sources such as Prometheus, Elasticsearch or a provider's monitoring service. One Grafana dashboard can show golden-signal panels from several sources side by side, which is why it appears so often next to Prometheus and the ELK stack.

Step 4: trace a request

Metrics say checkout is slow; a trace says why. OpenTelemetry describes a span as a single unit of work, with a name and timing data, and a distributed trace as the path of one request, made of one or more spans. The first span is the root span, representing the whole request from start to finish, and the spans beneath it are the calls it made. Laid out as a waterfall, the trace of an eight-second checkout might show the root span at 8 seconds and the payment call at 0.3 seconds. An inventory lookup takes 7.2 seconds, because it repeats a slow database query forty times. No single server's CPU would have revealed that. Tracing needs each service to pass the trace context along with its calls, which is what OpenTelemetry's libraries standardize across languages and providers.

Step 5: alert, triage and respond

An alert is a monitoring rule that interrupts a person, and the SRE book's tests for one are strict. Every page should be actionable, about an urgent condition that is user-visible or about to be, and novel rather than a known issue with a rote response. The reason is human: an on-call engineer can react with urgency only a few times a day before becoming fatigued. Lesser conditions belong in tickets or email for working hours.

Now change one input and watch the design fail. A team pages on every host above 80 percent CPU. Most nights forty pages arrive, almost none affect users, and the engineer starts acknowledging them unread. The SRE book describes exactly this pattern: a team spending large amounts of time triaging voluminous alerts to find the few actionable ones, and often missing the problems that actually affected users. Rewrite the rules around symptoms, such as an error rate above 2 percent for five minutes or latency beyond the promised threshold, and the pager falls quiet except when users are hurting.

When an alert does fire, triage comes first: confirm it, judge user impact and severity, and decide who must respond. Response then restores service before explaining it, by rolling back, failing over or adding capacity; tells affected people what is happening; and only afterward digs for the root cause and records what was learned. A short runbook linked from each alert, saying what the alert means and what to check first, turns a 3 AM page into a procedure.

The core of it: watch what users feel, keep the evidence where it survives and can be searched, and interrupt a human only for something urgent, actionable and new.

Common misconceptions

• "Monitoring CPU and memory is enough." Users experience latency and errors; host resources mostly explain saturation and should rarely page anyone.
• "Logs can stay on each server." Disposable instances take their logs with them, and incidents need logs from many systems searched together.
• "More alerts mean safer operations." Noisy, unactionable alerts train people to ignore pages, and real outages get lost among them.
• "Cloud logs are deleted automatically after a while." CloudWatch Logs, for example, keeps data indefinitely unless retention is set.

What to remember

• Logs say what happened, metrics say how much and how fast, and traces say where a request spent its time.
• The four golden signals are latency, traffic, errors and saturation.
• Collect structured logs, aggregate them centrally with synchronized clocks, and set retention deliberately.
• Prometheus scrapes labelled time series and records up for every target; Grafana queries and visualizes many data sources; the ELK stack ships, transforms, stores and visualizes logs.
• Page only for urgent, actionable, user-visible conditions; triage, restore service, communicate, then find and record the cause.

Sources

1. Beyer, B., Jones, C., Petoff, J., and Murphy, N. R. (Eds.). (2016). Monitoring distributed systems. In Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
2. OpenTelemetry Authors. (n.d.). Observability primer. OpenTelemetry documentation. opentelemetry.io
3. Kent, K., and Souppaya, M. (2006). Guide to computer security log management (NIST Special Publication 800-92). National Institute of Standards and Technology. csrc.nist.gov
4. Elastic. (n.d.). The Elastic Stack, Logstash and Beats. elastic.co (stack); elastic.co (Logstash); elastic.co (Beats)
5. Prometheus Authors. (n.d.). Overview, Installation and Jobs and instances. Prometheus documentation. prometheus.io (overview); prometheus.io (installation); prometheus.io (jobs)
6. Grafana Labs. (n.d.). Introduction to Grafana and Sign in to Grafana. Grafana documentation. grafana.com (introduction); grafana.com (sign in)
7. Amazon Web Services. (n.d.). Working with log groups and log streams (log retention). Amazon CloudWatch Logs User Guide. docs.aws.amazon.com

Four Pods at 90 percent

A Kubernetes Deployment runs 4 Pods behind a Service. A HorizontalPodAutoscaler watches them with a target of 50 percent average CPU utilization, and at the moment the Pods average 90 percent. What happens next? The Kubernetes documentation gives the rule exactly: the desired number of replicas is the current number multiplied by the ratio of the current metric to the target, rounded up. Four times 90 divided by 50 is 7.2, so the autoscaler asks for 8 Pods. When load later falls to an average of 20 percent across those 8, the same rule gives 3.2, so 4 Pods, but not at once: by default Kubernetes waits through a 300-second stabilization window before scaling down, and applies no such window when scaling up.

That small calculation raises the questions this lesson answers. Why add Pods rather than make each one bigger? Why hurry up and dawdle down? And what if CPU is not where the trouble is? The lesson covers CV0-004 objective 3.2: scaling types, horizontal and vertical, and scaling approaches, triggered by load, trend or event, scheduled, or manual.

Up or out

Kubernetes defines the two types in one sentence. Horizontal scaling means responding to increased load by deploying more Pods; vertical scaling would mean assigning more resources, such as memory or CPU, to the Pods already running. The same distinction applies to virtual machines: add instances behind a load balancer, or move one instance to a larger type.

The "software must" row decides more cases than any other. Horizontal scaling assumes that any instance can serve any request. If a web application keeps each user's shopping cart in its own memory, a request routed to a new instance finds no cart, so the fix is to move session state into a shared store, such as a cache or database, before scaling out.

When to scale: manual, scheduled, triggered

Trending deserves a word because it is the newest. AWS describes its predictive scaling as analyzing historical load data to detect daily or weekly patterns in traffic and forecasting future capacity needs, so capacity can be launched in advance of the forecast load. That matters whenever new capacity is slow to become useful. If instances take six minutes to boot and warm up, a purely reactive trigger always arrives six minutes late for a 9 AM surge, and a schedule or a forecast arrives on time.

Working the numbers: speed, bounds and patience

Three settings shape every autoscaler's behaviour. Bounds, a minimum and maximum, keep a tier from shrinking below what resilience needs, such as two instances in two zones, or growing past what the budget and quotas allow. Asymmetric timing makes scale-out fast and scale-in cautious: Kubernetes' 300-second scale-down window exists to restrict flapping, the expensive cycle of removing capacity during a brief dip and adding it back seconds later. And warm-up time, how long a new instance takes before it can serve, decides whether reactive scaling is fast enough at all.

Bottom line: scale out fast, scale in slowly, set bounds that protect both availability and budget, and use schedules or forecasts when capacity takes longer to start than users will wait.

The second run: a trigger that cannot see the problem

Now change one input. A fleet of workers processes jobs from a queue, and each job mostly waits on a slow remote API. The autoscaler watches CPU with a target of 70 percent. During a spike, 3,000 jobs pile up, yet the workers' CPU sits near 15 percent, because waiting uses almost no CPU. The trigger never fires, and jobs wait for hours.

The failure is the choice of metric, not the autoscaler. AWS's guidance for queue workers is to scale on backlog per instance, the number of waiting messages divided by the number of instances, with a target set to the backlog each instance can clear within an acceptable wait. If each job takes 2 seconds and no job should wait more than 60, each worker can carry a backlog of 30; 3,000 waiting jobs then call for 100 workers. In Kubernetes, KEDA, a Kubernetes-based event-driven autoscaler, drives scaling from the number of events needing to be processed and works alongside the Horizontal Pod Autoscaler. The general rule: trigger on the resource that is actually saturated, which is one of the four golden signals from Lesson 21.

Choosing, scenario by scenario

• A stateful database short of memory at month-end: vertical scaling, scheduled for a maintenance window because of the restart.
• A stateless API with unpredictable traffic: horizontal, triggered by load, with sensible bounds.
• Daily business-hours peaks and slow-booting instances: scheduled or predictive scaling, with load triggers as a backstop.
• Queue or event processing: event-triggered on backlog, possibly scaling to zero when idle.
• A one-off launch next Tuesday: manual or scheduled pre-scaling, with automatic scaling still enabled for surprises.

Common misconceptions

• "Vertical scaling happens without interruption." Changing an instance's size usually needs a restart; AWS requires the instance to be stopped first.
• "Autoscaling reacts instantly." Metrics are averaged over time, new capacity must boot, and scale-in waits on purpose.
• "CPU is the right trigger for everything." Queue workers and I/O-bound services saturate elsewhere; scale on backlog, requests or latency instead.
• "Any application can scale horizontally." Only if instances are interchangeable, which means keeping state outside them.

The takeaway

• Vertical scaling enlarges one server, usually with a restart and a size ceiling; horizontal scaling adds interchangeable instances and needs state kept outside them.
• Manual and scheduled scaling suit known events and cycles; triggered scaling reacts to load, forecasts trends or follows events waiting to be processed.
• The Kubernetes HPA computes replicas as current replicas times current metric over target, rounded up, scales up at once and scales down after a 300-second window.
• Pick the trigger that measures the real bottleneck, and scale ahead of demand when capacity starts slowly.

Sources

1. The Kubernetes Authors. (n.d.). Horizontal Pod Autoscaling and HorizontalPodAutoscaler walkthrough. Kubernetes documentation. kubernetes.io (concepts); kubernetes.io (walkthrough)
2. Amazon Web Services. (n.d.). Choose your scaling method, Predictive scaling for Amazon EC2 Auto Scaling and Scaling policy based on Amazon SQS. Amazon EC2 Auto Scaling User Guide. docs.aws.amazon.com (methods); docs.aws.amazon.com (predictive); docs.aws.amazon.com (queue)
3. Amazon Web Services. (n.d.). Change the instance type. Amazon EC2 User Guide. docs.aws.amazon.com
4. Microsoft. (n.d.). Overview of autoscale with Azure Virtual Machine Scale Sets. Microsoft Learn. learn.microsoft.com
5. KEDA Authors. (n.d.). KEDA: Kubernetes event-driven autoscaling. keda.sh
6. Kubernetes SIGs. (n.d.). Metrics Server [README]. github.com

A wrong answer with a replica in it

"We replicate the database to a standby in a second zone, continuously, so we do not need backups." It sounds like belt and braces. Now trace a Tuesday afternoon. At 14:02 an engineer runs a DELETE statement against the orders table and forgets the WHERE clause. Every row disappears. Replication does its job perfectly: within seconds the standby deletes every row too. Both copies agree, and both are empty.

The reasoning failed at one word. Replication keeps copies current, which protects availability when hardware, a zone or a server fails. It copies mistakes, corruption and ransomware encryption just as faithfully. A backup keeps copies of the past: a point in time you can return to, 14:01 in this case. Both are useful, and neither replaces the other. This lesson works through that misunderstanding and two more, and covers CV0-004 objective 3.3: backup types, locations, schedule, retention, replication, encryption, testing for recoverability and integrity, recovery types and recovery options.

Three backup types, traced through one week

Take a 500 GB database that changes by about 20 GB a day, with a full backup every Sunday. The three types differ in what each weekday's backup copies.

The table shows the trade. Incrementals are the cheapest to store and the quickest to take, but a restore needs the whole chain, and one damaged link breaks every later one. Differentials cost more storage as the week goes on but restore from two sets. Daily fulls are simplest to restore and most expensive to keep. Cloud snapshots usually behave as incrementals: AWS describes an EBS snapshot as an incremental backup that saves only the blocks changed since the most recent snapshot, while still letting you restore any snapshot on its own.

Key idea: replication copies the present, mistakes included; backups preserve the past. Incremental, differential and full backups trade storage against the number of sets a restore must read.

A second wrong answer: "our backups are off-site, they are in another bucket"

A company stores its backups in a storage bucket in the same cloud account and region as production. Attackers steal an administrator's credentials, encrypt production and then delete every backup, because the same credentials could. The backups were never off-site in the sense that matters: they shared production's blast radius.

CISA's long-standing advice is the 3-2-1 rule: keep three copies of important data, one primary and two backups, on two different kinds of media, with one copy stored off site. Translated to the cloud, on-site backups, in the same region, give fast routine restores, while the off-site copy belongs in another region and preferably another account with separate credentials. Immutability closes the last gap. AWS Backup Vault Lock, for example, stores backups as write-once, read-many: in compliance mode, once a short cooling-off period passes, neither users nor AWS can delete the backups before their retention ends, which is exactly the property ransomware operators cannot get around.

Three more settings complete the plan.

• Schedule. Derive it from the RPO in Lesson 4: nightly backups cannot meet a 15-minute RPO, which needs continuous log or snapshot protection.
• Retention. Decide how long each backup lives, from policy, law and how long problems take to notice. If corruption starts 45 days before anyone sees it and backups last 30, every copy is already corrupt; weekly and monthly copies kept longer prevent that.
• Encryption. Encrypt backups at rest and in transit, and keep the keys recoverable apart from the systems they protect. restic, the open-source tool in this lesson's lab, encrypts every file with AES-256 and authenticates it with a Poly1305-AES code; lose the password and the backup is unreadable by design.

A third wrong answer: "the job says success"

Every nightly backup job has reported success for a year. Then a server dies, and the restore fails: a configuration change months ago excluded the data directory, or the media is damaged, or nobody knows the procedure. A successful job proves that something was written, not that it can be brought back. NIST's contingency planning guidance builds testing into the backup control itself: test backup information to verify media reliability and information integrity, and exercise recovery, including restoring from backup, so that staff are familiar with it.

Testing has two layers. Integrity checks that stored data is undamaged and complete; restic's check command verifies the repository's structure, and with --read-data it reads every stored byte and verifies its authentication code. Recoverability checks that a real restore produces a working system: restore into a test environment, start the application, confirm the data, and time the whole process against the RTO. A restore that works but takes nine hours fails a four-hour RTO as surely as one that fails outright.

Getting it back: in-place or parallel, bulk or granular

Return to the 14:02 deletion. An in-place bulk restore of last night's backup would bring the orders back and destroy every other change made since midnight. The better path is parallel: restore to a point just before 14:02 in a new database, then copy the orders table back. AWS's point-in-time restore for RDS works this way by design, creating a new database instance without modifying the source. For a single deleted spreadsheet on a 2 TB file share, a granular restore of one file avoids overwriting a day of everyone else's work.

Common misconceptions

• "Replication is a backup." It copies deletions, corruption and ransomware encryption instantly. It protects availability, not history.
• "Incrementals restore fastest because they are small." A restore needs the full backup plus every incremental after it; differentials need only two sets.
• "A backup in another bucket is off-site." Off-site means outside production's blast radius: another region or account, ideally immutable.
• "A successful backup job means a working backup." Only an integrity check and a timed test restore prove that.

Recap

• Full backups copy everything; incrementals copy changes since the last backup; differentials copy changes since the last full. Restores need one set, the whole chain, or two sets respectively.
• Follow 3-2-1: on-site copies for quick restores, an off-site copy in another region or account, and immutability against ransomware.
• Set the schedule from the RPO and retention from policy and detection time; encrypt backups and keep their keys recoverable.
• Test integrity and recoverability, timing restores against the RTO.
• Restore in place or in parallel, in bulk or granularly, choosing the option that recovers the loss without destroying good data.

Sources

1. Ruggiero, P., and Heckathorn, M. A. (2012). Data backup options. US-CERT, Carnegie Mellon University. cisa.gov (PDF)
2. Swanson, M., Bowen, P., Phillips, A. W., Gallup, D., and Lynes, D. (2010). Contingency planning guide for federal information systems (NIST Special Publication 800-34 Revision 1), including control CP-9. National Institute of Standards and Technology. csrc.nist.gov
3. Amazon Web Services. (n.d.). Amazon EBS snapshots. Amazon EBS User Guide. docs.aws.amazon.com
4. Amazon Web Services. (n.d.). AWS Backup Vault Lock. AWS Backup Developer Guide. docs.aws.amazon.com
5. Amazon Web Services. (n.d.). Restoring a DB instance to a specified time. Amazon RDS User Guide. docs.aws.amazon.com
6. restic authors. (n.d.). restic documentation: working with repositories, restoring from backup, removing snapshots and design references. restic.readthedocs.io (check); restic.readthedocs.io (restore); restic.readthedocs.io (encryption)

A year's notice

On 8 December 2020 the CentOS Project announced that it was shifting its focus to CentOS Stream. CentOS Linux 8, the free rebuild of Red Hat Enterprise Linux 8 that ran on a great many servers and cloud images, would end at the close of 2021. The project's end-of-life page later spelled out what that meant in practice. After 31 December 2021 there would be no more updates to CentOS Linux 8; its packages would be moved off the mirrors to an archive; and nobody would be migrated to anything automatically.

Teams running CentOS Linux 8 had a year to do every part of life-cycle management at once. They had to keep patching, test a replacement, move to a new major version or another distribution, carry their data across and switch off the old systems cleanly. Following that year is a good way through CV0-004 objective 3.4, managing the life cycle of cloud resources: patches, updates both major and minor, testing, ephemeral and persistent data, and decommissioning at end of life and end of support.

Patches, minor updates and major updates

Software changes come in sizes, and Semantic Versioning gives the sizes names. In a version number such as 2.7.3, the specification says you increment the major version when you make incompatible API changes, the minor version when you add functionality in a backward-compatible way, and the patch version when you make backward-compatible bug fixes. Operating systems and cloud services do not all follow SemVer strictly, but the three sizes map onto the decisions an operations team faces.

NIST's Special Publication 800-40 Revision 4, from April 2022, describes enterprise patch management as identifying, prioritizing, acquiring, installing and verifying the installation of patches, updates and upgrades across an organization. Its central argument is about attitude. Business owners often see patching as lost productivity, because it needs scheduled downtime and can cause more if something goes wrong. NIST's answer is that patching should be treated as a standard cost of doing business, like preventive maintenance on a fleet of vehicles, which avoids costly breakdowns. NIST also notes the case that CentOS users faced: when software reaches end of life, a patch for its next vulnerability will never be released.

The point: a patch fixes, a minor update adds compatibly, a major update breaks compatibility. Patching is preventive maintenance with a schedule, not an emergency you wait for.

Testing before production

Every change can break something, so changes reach production only after a test environment has shown they do not. Infrastructure as code from Lesson 17 makes a faithful staging environment cheap: the same code, with different variables, builds a copy of production to patch first. Automated tests then check that the application still works. The deployment strategies from Lesson 20 limit the damage if testing missed something. A canary exposes a new image to a few users first, and a rolling update stops if new instances fail their health checks. Two more habits close the loop. Take a snapshot or backup before a risky change, so rollback is a restore rather than a rebuild. Afterward, verify that the patch is actually installed, the last step in NIST's definition and the one most often skipped.

Patch in place, or replace

There are two ways to apply a change to a cloud workload. You can patch the running instances in place, as on-premises servers have always been patched. Or you can build a new machine or container image with the change already applied and replace the running instances with new ones, discarding the old. Replacement is the cloud's natural habit, because it makes every instance identical and every change repeatable, and it is how container images and autoscaling groups are meant to be updated.

Replacement works only if the data is in the right place. Anything stored on the instance itself is lost when it is replaced, so the decision depends on which data is ephemeral and which is persistent.

A team whose CentOS servers kept their application data on the root disk had to copy it somewhere before rebuilding on a new operating system. A team that had always kept data on separate volumes or managed services could build new images on the new distribution, attach the same data and switch over.

End of support, end of life, and switching off

Every product reaches an end. CompTIA's objective names two milestones, and vendors use the words in slightly different ways, so read each vendor's definitions. End of life generally means the product is no longer developed or sold; end of support means the vendor stops providing fixes and help. Microsoft's lifecycle pages state what end of support means for its products: no new security updates, non-security updates, assisted support or online content updates. Its 2023 list includes Windows Server 2012 components retiring on 10 October 2023. Cloud platforms have life cycles too: the Kubernetes project maintains release branches only for the three most recent minor releases, each with roughly a year of patch support, which makes upgrading a managed cluster an annual chore rather than an option. Lesson 17 met another example, Google Cloud Deployment Manager, whose support ended on 31 March 2026, and Lesson 33 shows what such retirements do to deployments that still depend on them.

When a product reaches its end and cannot be upgraded at once, NIST's patch guidance describes the remaining option for unpatchable assets: isolate them and add compensating controls until they can be replaced. That buys time; it does not make the system safe.

Decommissioning, switching a resource off for good, is a procedure, not a delete button.

1. Confirm dependencies. Check monitoring, DNS, firewall rules and other systems for anything still using the resource.
2. Keep what must be kept. Archive data and logs required by retention policy or law (Lesson 26), and migrate data that moves to the replacement.
3. Revoke access. Remove credentials, keys, service accounts and firewall openings that existed only for this resource.
4. Delete everything attached. Volumes, snapshots, elastic IP addresses, DNS records and load balancer entries can outlive the instance and keep billing (Lesson 30 calls forgotten ones zombies).
5. Sanitize. NIST SP 800-88 Revision 2, from September 2025, defines media sanitization as a process that makes access to target data infeasible for a given level of effort. You cannot wipe a provider's physical disks, so NIST notes that for cloud storage cryptographic erase, destroying the keys that encrypted the data, may be the only viable purge technique. Encrypting cloud data with keys you control is what makes that option possible.
6. Update the records. Remove the resource from inventories and documentation so nobody patches or pays for a ghost.

Common misconceptions

• "A minor update is as safe as a patch." Minor updates add features and occasionally change behaviour; they need testing in staging like any other change.
• "Unsupported software keeps working, so it is fine." It keeps running, but its next vulnerability will never be patched.
• "Replacing an instance is the same as rebooting it." Replacement discards everything ephemeral; only data on persistent storage carries over.
• "Deleting a VM decommissions it." Snapshots, volumes, addresses, DNS records, credentials and data obligations remain until someone handles them.

What to carry forward

• Patches fix, minor updates add compatibly, major updates break compatibility; patch management identifies, prioritizes, acquires, installs and verifies them as routine preventive maintenance.
• Test in staging built from the same code, roll out with canary or rolling strategies, snapshot first, and verify installation afterward.
• Replacing instances from patched images is the cloud's natural update method, provided persistent data lives off the instance.
• End of support ends fixes; end of life ends the product. Plan upgrades ahead, isolate what cannot yet move, and decommission with care, using cryptographic erase for cloud data.

Sources

1. CentOS Project. (2020, December 8). CentOS Project shifts focus to CentOS Stream, and (n.d.) CentOS Linux EOL. blog.centos.org; centos.org
2. Preston-Werner, T. (n.d.). Semantic versioning 2.0.0. semver.org
3. OpenSSL Library. (n.d.). Versioning policy. openssl-library.org
4. Souppaya, M., and Scarfone, K. (2022). Guide to enterprise patch management planning: Preventive maintenance for technology (NIST Special Publication 800-40 Revision 4). National Institute of Standards and Technology. doi.org
5. Microsoft. (n.d.). Products reaching end of support for 2023. Microsoft Learn. learn.microsoft.com
6. The Kubernetes Authors. (n.d.). Releases. Kubernetes documentation. kubernetes.io
7. National Institute of Standards and Technology. (2025). Guidelines for media sanitization (NIST Special Publication 800-88 Revision 2). csrc.nist.gov
8. Canonical. (n.d.). Ubuntu release cycle. ubuntu.com

165 organizations and one missing factor

In April 2024 Mandiant, Google's incident response firm, began investigating stolen database records that turned out to come from one company's account on Snowflake, a cloud data platform. Within weeks the case had grown into a campaign, and Mandiant and Snowflake notified approximately 165 potentially exposed organizations. Mandiant found no evidence of a breach of Snowflake's own enterprise environment. Every incident it handled traced back to customer credentials stolen earlier by infostealer malware on unrelated computers; the earliest infection it linked to a credential used in the campaign dated from November 2020. The accounts did not have multifactor authentication enabled, so a valid username and password were all an attacker needed. Many of the stolen credentials had never been rotated, and the affected accounts had no network allow lists restricting where sign-ins could come from.

Nothing in that campaign needed a software vulnerability. It needed only identities that anyone holding a password could use. Identity and access management, IAM, is the set of controls that decides who can act in a cloud environment, how they prove it, what they may do and how their actions are recorded. This lesson builds those controls step by step and covers CV0-004 objective 4.3: securing access to the cloud management environment and to cloud resources, authentication models, authorization models, and accounting through an audit trail. Security+ Certification Prep (cs-secplus, CS 316) treats identity in more depth; this lesson stays with the cloud.

Four questions for every request

The cloud sharpens all four. Creating a network, resizing a database or deleting a storage bucket is an API call, whether it starts in a browser or a script. So anyone who can authenticate to the provider's API with enough permissions can do from anywhere what an administrator could once do only from inside the data center.

Step 1: list every way in

Start with an inventory, because an access path nobody remembers is one nobody secures. CompTIA separates two kinds. The management environment is where accounts, networks and services are configured; the resources are the servers, databases and storage that run the workload.

A note on terms: the acronym list in CompTIA's objectives expands CLI as Common Language Infrastructure, which is a .NET term. In objective 4.3 it plainly means the command-line interface, the provider tools named above, and that is how this course uses it.

One identity deserves its own line in the inventory: the account's original owner, which AWS calls the root user. It can do anything, including closing the account, so it deserves the strongest MFA available, no access keys and no everyday use. AWS's IAM best practices list protecting root user credentials alongside requiring MFA and using temporary credentials.

Step 2: decide how identities prove who they are

SAML, the Security Assertion Markup Language, is an OASIS standard that defines an XML-based framework for exchanging security information, in the form of assertions, between an identity provider and a service provider such as a cloud console. It dominates enterprise single sign-on. OpenID Connect is, in its specification's own words, a simple identity layer on top of the OAuth 2.0 protocol. It lets an application verify who the user is from the authentication an authorization server performed, through an ID token represented as a JSON Web Token. It is the newer, JSON-based option, common in web and mobile apps and in workload federation. GitHub's documentation, for example, describes how OpenID Connect lets a pipeline exchange its own token for a short-lived access token from a cloud provider, so no long-lived cloud secret has to be stored in the pipeline at all.

For MFA, NIST's Digital Identity Guidelines, Special Publication 800-63-4, finalized in July 2025, classify authenticators by factor. A password is something you know, and a code card, a phone that receives codes or a security key is something you have. A biometric is something you are, though NIST does not accept a biometric as an authenticator by itself, only alongside a physical one. The guidelines draw a line that matters more every year. Authenticators whose output is typed in by hand, such as one-time codes, are not phishing-resistant, because a fake site can relay the code to the real one while it is still valid. Phishing resistance requires cryptographic authentication bound to the session or to the genuine site's name; WebAuthn, used by FIDO2 security keys and passkeys, is NIST's example. At NIST's second assurance level, applications must offer a phishing-resistant option.

Step 3: grant the least access that works

Authentication settles who; authorization settles what. The aim is the principle of least privilege, giving each identity only the permissions its job needs, and the objective names four models for getting there.

OAuth 2.0 needs one clarification. RFC 6749 describes it as an authorization framework that lets an application obtain limited access to an HTTP service, either on behalf of a resource owner or on its own behalf. An access token says what its bearer may do, not who the user is, and that is the gap OpenID Connect fills.

The providers implement the same ideas with different nouns.

Least privilege is a habit rather than a setting. Start from the provider's managed roles and tighten them: AWS's best practices suggest generating least-privilege policies from actual access activity and regularly removing unused users, roles, permissions and credentials. Give workloads roles and temporary credentials instead of stored keys. The credentials used in the Snowflake campaign were still valid, in some cases, years after they were stolen.

Step 4: secure the paths to resources

• SSH: use key pairs, such as ed25519 keys, and disable password sign-in. The private key never leaves the administrator's machine; only the public key goes on the server.
• RDP: never expose port 3389 to the internet. Keep Network Level Authentication on, which Microsoft describes as making users authenticate before a remote session is established, and reach Windows servers through a gateway or bastion.
• APIs: authenticate every call with short-lived tokens or signed requests, and scope each token to what its caller needs.
• Bastion host: NIST defines one as a special-purpose computer on a network specifically designed and configured to withstand attacks. In a cloud network it is a hardened jump box in a public subnet and the only machine reachable from outside; security groups allow SSH or RDP to the private servers only from the bastion, and OpenSSH's jump-host option, ssh -J, hops through it in one command.

Each provider also sells a way to reach private servers without running a bastion of your own.

Step 5: keep an audit trail

Accounting answers the question Google's audit logging documentation puts plainly: who did what, where and when. Every provider records control-plane activity out of the box, but the defaults differ, so learn them before you need them.

Protect the trail as carefully as the systems it records. Send a copy to a separate, locked-down account or project, so that an intruder with administrator rights in one account cannot erase the evidence, and set retention to what policy and law require (Lessons 21 and 26). An audit trail is a detection tool as well as a record: a sign-in from an unexpected country, a burst of permission changes or a new access key deserves an alert (Lesson 30).

Why this matters: in the cloud, identity is the front door to everything. In the Snowflake campaign, MFA would have stopped sign-ins that needed only a password. Rotation or short-lived credentials would have expired the stolen ones, and network allow lists would have refused sign-ins from unexpected places. Each of the five steps above closes one of those doors.

Common misconceptions

• "OAuth 2.0 tells an application who the user is." It grants limited access through tokens; OpenID Connect adds the ID token that identifies the user.
• "Any second factor stops phishing." Codes typed by hand can be relayed by a fake site. NIST counts only cryptographic authenticators bound to the session or the site, such as FIDO2 security keys and passkeys, as phishing-resistant.
• "A bastion host makes the servers behind it safe." It narrows the way in. It still needs hardening, MFA, logging and patching, and managed services can remove it altogether.
• "A key in a private repository is safe." Long-lived keys leak through laptops, logs and malware and stay valid until someone revokes them; roles, federation and short-lived credentials leave nothing durable to steal.

What you now know

• IAM answers four questions for every request: who is acting, how they proved it, what they may do and what they did.
• Inventory every path in, for management (web portal, CLI, API and SDK) and for resources (service APIs, SSH, RDP and bastion hosts), and guard the root or owner account above all.
• Authentication can rest on local users, a directory, federation through SAML or OpenID Connect, or tokens; MFA should be phishing-resistant wherever possible.
• Authorization uses roles, groups, owner discretion or OAuth 2.0 scopes; aim for least privilege and temporary credentials.
• Audit trails record who did what. Know each provider's defaults, keep logs longer where policy requires, and protect them from the people they record.

Sources

1. Mandiant. (2024, June 10). UNC5537 targets Snowflake customer instances for data theft and extortion. Google Cloud Blog. cloud.google.com
2. National Institute of Standards and Technology. (2025). Digital identity guidelines (NIST Special Publication 800-63-4) and Authentication and authenticator management (SP 800-63B-4). csrc.nist.gov; pages.nist.gov
3. OASIS. (2008). Security Assertion Markup Language (SAML) V2.0 technical overview. docs.oasis-open.org
4. OpenID Foundation. (n.d.). OpenID Connect Core 1.0. openid.net
5. Hardt, D. (Ed.). (2012). The OAuth 2.0 authorization framework (RFC 6749). Internet Engineering Task Force. rfc-editor.org
6. National Institute of Standards and Technology. (n.d.). Role based access control project, and glossary entries bastion host and discretionary access control. csrc.nist.gov (RBAC); csrc.nist.gov (bastion host); csrc.nist.gov (DAC)
7. Amazon Web Services. (n.d.). Security best practices in IAM, Policy evaluation logic, What is AWS CloudTrail? and AWS Systems Manager Session Manager. docs.aws.amazon.com (IAM); docs.aws.amazon.com (evaluation); docs.aws.amazon.com (CloudTrail); docs.aws.amazon.com (Session Manager)
8. Microsoft. (n.d.). What is Azure role-based access control?, What is Azure Bastion?, Activity log in Azure Monitor and Allow access to your PC with Remote Desktop. Microsoft Learn. learn.microsoft.com (RBAC); learn.microsoft.com (Bastion); learn.microsoft.com (activity log); learn.microsoft.com (Network Level Authentication)
9. Google Cloud. (n.d.). IAM overview, Cloud Audit Logs overview and Using IAP for TCP forwarding. docs.cloud.google.com (IAM); docs.cloud.google.com (audit logs); docs.cloud.google.com (IAP)
10. GitHub. (n.d.). OpenID Connect. GitHub Docs. docs.github.com; The Kubernetes Authors. (n.d.). Using RBAC authorization. kubernetes.io; OpenBSD. (n.d.). ssh(1). man.openbsd.org

1.2 billion euros for a transfer

On 22 May 2023 Ireland's Data Protection Commission announced the end of an inquiry into Meta Platforms Ireland, the company responsible for Facebook in the European Union. The Commission found that Meta Ireland had broken the GDPR's rules on international transfers. It kept sending EU and EEA users' personal data to the United States after the EU's Court of Justice ruled in July 2020, in the case known as Schrems II.

Meta had relied on the European Commission's standard contractual clauses plus supplementary measures of its own. The Irish regulator found that these did not address the risks to people's fundamental rights that the court had identified. The penalty was an administrative fine of 1.2 billion euros and an order to suspend future transfers within five months. A further order required Meta to stop the unlawful processing, including storage, in the US of the EU users' data within six months.

Seven weeks later, on 10 July 2023, the European Commission adopted an adequacy decision for a new EU-U.S. Data Privacy Framework, concluding that the United States ensures an adequate level of protection for personal data sent to US companies that participate in it. The legal ground had moved twice in three years. Questions about where data may live, whose law reaches it and how long it may be kept are not paperwork at the edge of cloud architecture; they decide which regions, services and contracts are allowed at all.

This lesson compares the concepts and standards in CV0-004 objective 4.2: data sovereignty, ownership, locality and classification; data retention, including litigation holds and contractual and regulatory retention; and the industry standards SOC 2, PCI DSS, ISO 27001 and the Cloud Security Alliance's frameworks. It explains the rules so that you can design for them; it is not legal advice, and real decisions belong with an organization's lawyers.

Sovereignty, locality and residency compared

The concepts are easy to confuse because locality is one tool for sovereignty. Keeping data in a Frankfurt region satisfies a locality rule, but if the provider is subject to another country's laws, those laws may still reach the data. The US Department of Justice's 2019 white paper on the CLOUD Act explains what the act does. It makes explicit in US law that a company subject to a country's jurisdiction can be required to produce data it controls, regardless of where it is stored. The paper also notes that many other governments can require the same of companies within their own jurisdiction.

Providers state the limits of their promises carefully. AWS says customers choose the regions where their content is stored. It will not move or replicate that content outside those regions, it adds, with two exceptions. One is when that is necessary to provide services the customer initiated; the other is complying with the law or a binding order of a governmental body. Microsoft describes its EU Data Boundary as a geographically defined boundary for its enterprise online services, including Azure and Microsoft 365. Within it, Microsoft has committed to store and process customer data and personal data, subject to limited, documented transfers. Read such commitments for what they cover and what they exclude.

Locality must cover every copy. Data replicated to a second region for disaster recovery (Lessons 4 and 23), cached at a CDN's edge (Lesson 7) or shipped in logs to a central account (Lesson 21) can each carry it out of the permitted area.

Ownership and classification

Data ownership answers who has the rights and who carries the duties. Between customer and provider, the provider's terms settle the rights: AWS, for example, states that customers own their customer content and select which services can process, store and host it. Under the GDPR the duties come with roles. The controller determines the purposes and means of processing personal data; the processor processes it only on the controller's behalf, and the European Commission names cloud storage as a typical processor activity. The contract between them must say, among other things, what happens to the data when the contract ends. Inside an organization, a named data owner, usually a business leader rather than an IT administrator, decides who may use a data set. The owner also sets its classification and how long it is kept, and administrators act as custodians who carry those decisions out.

Data classification sorts data by the harm its exposure, corruption or loss would cause, so that controls can be matched to it. NIST's FIPS 199 rates the potential impact of a loss of confidentiality, integrity or availability as low, moderate or high, and many companies use labels like these.

Classification drives everything else in this lesson: where data may live, who may see it, whether it must be encrypted and how long it is kept. At cloud scale, discovery is partly automated. Amazon Macie discovers sensitive data in S3 using machine learning and pattern matching, Google Cloud's Sensitive Data Protection discovers, classifies and de-identifies sensitive data, and Microsoft Purview applies sensitivity labels that can encrypt and mark content.

Retention: three drivers compared

Regulations can set a minimum, a maximum or both. HIPAA's six years is a floor; the GDPR's storage limitation principle is a ceiling, which the European Commission summarizes as storing personal data for the shortest time possible. Contracts add floors and ceilings of their own, and the hardest part is usually the copies: a promise to delete within 30 days is broken by a backup kept for 90.

A litigation hold overrides both. Under Rule 37(e) of the US Federal Rules of Civil Procedure, a court can impose measures when electronically stored information that should have been preserved in the anticipation or conduct of litigation is lost because a party failed to take reasonable steps to preserve it. The duty therefore begins when litigation is anticipated, not when papers arrive, so automated deletion of relevant data must stop as soon as counsel says so. Cloud storage supports this directly. AWS describes an S3 Object Lock legal hold as giving the same protection as a retention period but with no expiration date, remaining until someone explicitly removes it. Azure's immutable blob storage keeps data under a legal hold immutable until the hold is explicitly cleared. Google Cloud Storage offers temporary holds that can be used for regulatory or legal purposes.

What matters here: sovereignty is whose law applies, locality is where the bytes sit, ownership is who decides, classification is how much protection the data needs, and retention is how long it must, or may, exist. A cloud design is compliant only when every copy of the data satisfies all five.

Industry standards compared

SOC 2 reports come in two types. Microsoft describes its SOC 2 Type 2 report as evaluating whether controls are designed appropriately, were in operation on a specified date and operated effectively over a specified period. A Type 1 report stops at design on a single date. AWS issues SOC 2 reports twice a year, each covering the previous 12 months. ISO/IEC 27001 is certified rather than attested: AWS lists certifications for ISO/IEC 27001:2022 together with the cloud-specific codes of practice ISO/IEC 27017 and 27018, issued by independent third-party auditors. The CSA's frameworks are the cloud specialists' contribution, and the STAR Registry is public, so a buyer can read a provider's answers before any sales call.

What a provider's certificate covers

Every large provider publishes its evidence. AWS customers download reports from AWS Artifact; Microsoft's Service Trust Portal publishes audit reports produced by external auditors; Google Cloud offers its certificates through its Compliance Reports Manager. This evidence covers the provider's half of the shared responsibility model from Lesson 2, and nothing more. Google's ISO/IEC 27001 page says it plainly: a customer organization will have to seek out and obtain its own certification, although it can build on the provider's. A shop that runs its checkout on a provider with a PCI DSS attestation still has to assess its own handling of card data. Likewise, a company whose provider holds a SOC 2 report still needs controls over its own accounts, configurations and data.

Common misconceptions

• "Choosing an EU region makes us GDPR-compliant." Region choice handles locality only. The GDPR also governs purposes, transfers, retention and people's rights, and other countries' laws may still reach the provider.
• "Our provider is certified, so we are compliant." Its certificate covers its controls; yours need their own assessment.
• "Keeping everything forever is the safe option." The GDPR sets a ceiling, contracts may require deletion, and every extra year adds breach exposure and cost.
• "A litigation hold can wait until the lawsuit is filed." The duty to preserve begins when litigation is reasonably anticipated.

The short version

• Sovereignty is whose law governs data and can compel access; locality is where every copy physically sits; residency is where the organization chose or promised to keep it.
• Ownership settles who decides. Under the GDPR, controllers decide, and processors such as cloud providers act on their behalf under contract.
• Classification sets the level of protection, and the other decisions follow from it.
• Retention comes from regulations, contracts and litigation holds; holds override deletion until explicitly released, and every rule must reach replicas and backups.
• SOC 2, PCI DSS, ISO/IEC 27001 and CSA STAR each prove different things, and a provider's attestation covers only the provider's side.

Sources

1. Data Protection Commission. (2023, May 22). Data Protection Commission announces conclusion of inquiry into Meta Ireland. dataprotection.ie
2. European Commission. (2023, July 10). Data protection: European Commission adopts new adequacy decision for safe and trusted EU-US data flows (IP/23/3721); and (n.d.) Principles of the GDPR and Application of the regulation. ec.europa.eu; commission.europa.eu (principles); commission.europa.eu (controllers and processors)
3. US Department of Justice. (2019). Promoting public safety, privacy, and the rule of law around the world: The purpose and impact of the CLOUD Act [White paper]. justice.gov
4. Amazon Web Services. (n.d.). Data privacy FAQ, Locking objects with Object Lock, SOC FAQs, ISO/IEC 27001:2022 compliance and What is Amazon Macie? aws.amazon.com (privacy); docs.aws.amazon.com (Object Lock); aws.amazon.com (SOC); aws.amazon.com (ISO 27001); docs.aws.amazon.com (Macie)
5. Microsoft. (n.d.). What is the EU Data Boundary?, Overview of immutable storage for blob data, SOC 2 Type 2, Learn about sensitivity labels and Get started with the Microsoft Service Trust Portal. Microsoft Learn. learn.microsoft.com (EU Data Boundary); learn.microsoft.com (immutable storage); learn.microsoft.com (SOC 2); learn.microsoft.com (sensitivity labels); learn.microsoft.com (Service Trust Portal)
6. Google Cloud. (n.d.). ISO/IEC 27001, Object holds and Sensitive Data Protection overview. cloud.google.com (ISO 27001); docs.cloud.google.com (object holds); docs.cloud.google.com (Sensitive Data Protection)
7. National Institute of Standards and Technology. (2004). Standards for security categorization of federal information and information systems (FIPS 199). csrc.nist.gov
8. 45 C.F.R. section 164.316 (security standards: policies, procedures and documentation requirements). ecfr.gov; Federal Rules of Civil Procedure, Rule 37. Legal Information Institute. law.cornell.edu
9. PCI Security Standards Council. (n.d.). PCI DSS, and (2024) Just published: PCI DSS v4.0.1. pcisecuritystandards.org; blog.pcisecuritystandards.org
10. AICPA and CIMA. (n.d.). System and Organization Controls: SOC suite of services. aicpa-cima.com; Cloud Security Alliance. (n.d.). Cloud Controls Matrix and CAIQ and STAR. cloudsecurityalliance.org (CCM); cloudsecurityalliance.org (STAR)

The problem: a fix that never arrived

In March 2017 a flaw in Apache Struts 2, a widely used Java web framework, became public. The National Vulnerability Database record for it, CVE-2017-5638, says that the Jakarta Multipart parser in Struts 2.3.x before 2.3.32 and 2.5.x before 2.5.10.1 let remote attackers execute arbitrary commands through a crafted HTTP header, and that it was exploited in the wild that same month. Fixed versions were available. The flaw was about as serious as a flaw can be, and the cure was an upgrade.

The US Government Accountability Office's 2018 report on what happened next at Equifax, report GAO-18-559, reads like a checklist of everything that can go wrong. Equifax received a notice about the vulnerability from the US Computer Emergency Readiness Team in March 2017 and circulated it to its system administrators, but the recipient list was out of date, so the people responsible for patching the affected system never received it. A network scan a week later did not detect the vulnerability on the company's online dispute portal. Beginning on 13 May 2017, attackers used the flaw to get into that portal, disguised their activity and removed data in small increments over encrypted web connections. A tool that should have inspected that traffic was blind because a digital certificate had expired about ten months earlier, and because the databases were not segmented, the attackers could reach far more than the portal's own data. Security staff found the intrusion on 29 July 2017, about 76 days after it began. The attackers had accessed personal information of at least 145.5 million people.

The patch existed. What failed was the process that should have carried it to the right server: knowing what to scan, finding the flaw, judging its urgency and fixing it, then checking the fix. That process is vulnerability management, and it is CV0-004 objective 4.1: its steps of scanning scope, identification, assessment and remediation, and the CVE system that names what is being managed.

Why the problem is hard

The first difficulty is volume. MITRE reports that the CVE list grew from 321 records in 1999 to more than 240,000 by October 2024, produced by over 400 CVE Numbering Authorities in 40 countries. The second is speed. When CISA created its catalog of known exploited vulnerabilities in 2021, it noted that fewer than 4 percent of all CVEs had been publicly exploited, but that among those, 42 percent were used on the day of disclosure, 50 percent within two days and 75 percent within 28 days. A team cannot fix everything at once, and it cannot wait to fix the few that matter.

CVE: one name for one flaw

Common Vulnerabilities and Exposures, CVE, is a program that MITRE launched in 1999 and that CISA now sponsors. It gives each publicly disclosed vulnerability one identifier in the form CVE-year-number. The identifiers are assigned by CVE Numbering Authorities, or CNAs: software vendors, open-source projects, researchers and national response teams. A CVE ID is a name, not a verdict. It lets a scanner, a vendor advisory, a patch note and a ticket all refer to exactly the same flaw.

Databases then add analysis to the name. NIST's National Vulnerability Database records, for example, the affected products and a CVSS score. For CVE-2017-5638 the NVD gives a CVSS 3.1 base score of 9.8, Critical, with the vector AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H.

CVSS is maintained by FIRST, the Forum of Incident Response and Security Teams; version 4.0, first published on 1 November 2023, groups its metrics into Base, Threat, Environmental and Supplemental. The Base score reflects a vulnerability's intrinsic characteristics, constant over time, and assumes a reasonable worst-case impact. The qualitative scale is None (0.0), Low (0.1 to 3.9), Medium (4.0 to 6.9), High (7.0 to 8.9) and Critical (9.0 to 10.0). Many databases still publish 3.1 vectors like the one above, so expect to read both versions.

Step 1: set the scanning scope

A scan finds problems only in what it covers. Equifax's patch went out across the company, but the portal running the vulnerable framework was not identified as needing it. Scope therefore starts with an inventory, and in a cloud environment the inventory changes every hour.

• Compute: virtual machines and the images they boot from, including stopped ones that will start again.
• Containers: images in registries and images running in clusters; a stored image grows new vulnerabilities while it sits unchanged.
• Functions and code: serverless packages and application dependencies, ideally listed in a software bill of materials (SBOM).
• Exposure: which of these can be reached from the internet, because exposure changes urgency.

Scope is also a matter of responsibility and permission. Under the shared responsibility model from Lesson 2, the provider fixes the platform beneath a managed service, while the customer finds and fixes flaws in what it manages, such as guest operating systems in IaaS, container images and code. Scan only what you own. AWS lets customers run security assessments of their own resources without prior approval for the services it lists, but prohibits denial-of-service testing; Microsoft's rules of engagement allow port scanning and vulnerability assessments on your own Azure virtual machines; Google says you do not need to contact it before testing your own projects, provided the tests affect only your projects and follow its acceptable use policy.

Step 2: identify

Scanners identify vulnerabilities by working out which software versions are present and matching them against vulnerability data. An authenticated or agent-based scan reads installed packages directly and sees far more than an unauthenticated network scan, which sees only what an outsider sees. Image scanners read the package databases inside container image layers. Open-source examples include Grype and Trivy, and each provider sells a managed service.

Identification must be continuous, because new CVEs are published every day for software you have not touched. Remember, too, that scanners are software with their own supply chain. According to the project's own security advisory, on 19 March 2026 attackers used compromised credentials to publish a malicious release of the Trivy scanner, v0.69.4, and to force-push 76 of the 77 version tags of its GitHub Action to credential-stealing code. CISA added the incident, CVE-2026-33634, to its known exploited catalog on 26 March. Users who referenced Trivy's container images by digest were not affected, which is why this course pins tool versions and, in pipelines, digests (Lessons 28 and 31).

Step 3: assess

Assessment decides what to fix first. Severity is only one input.

CISA's own rules show how the signals combine. In June 2026 it replaced the directive that created the KEV catalog, BOD 22-01, with BOD 26-04, which tells US federal agencies to set remediation deadlines from four questions: is the asset publicly exposed, is the vulnerability in the KEV catalog, can an adversary automate its exploitation, and would exploitation give partial or total control? Deadlines range from three days, some with a forensic check for compromise, through 14 and 60 days, to "fix on system upgrade", meaning at the asset's next scheduled major upgrade or rebuild. The directive binds only federal agencies, but its logic suits anyone: exposed, exploited and automatable flaws first.

Apply it to 2017. CVE-2017-5638 scores 9.8, has been in the KEV catalog since November 2021 with known use in ransomware campaigns, and in November 2026 its EPSS score was above 0.99. On an internet-facing portal it belongs at the top of any list; on an isolated test machine it would still need fixing, but not before the portal.

Step 4: remediate and verify

A remediation is finished only when a rescan shows the vulnerable version is gone, which is the last step NIST includes in patch management and the one Equifax's process did not reach. Track the time from detection to verified fix by severity and exposure; that number says more about a program than the count of open findings.

Remember: the four steps fail independently. A perfect scanner cannot find what is out of scope, a perfect score cannot prioritize what nobody assesses, and a patch that nobody verifies may never have reached the server that mattered.

Common misconceptions

• "The highest CVSS score always goes first." CVSS measures severity; exploitation evidence from the KEV catalog, EPSS and the asset's exposure decide urgency.
• "We scanned, so we are clean." A scan covers only its scope and can miss what it does cover; Equifax's scan missed the portal.
• "Containers are rebuilt, so they never need patching." Rebuilding helps only from a patched base, and images that sit unchanged gather new CVEs.
• "The provider patches everything in the cloud." It patches its side of the shared responsibility line; guest operating systems, images and code are yours.

Putting it together

• Vulnerability management is a cycle: set the scanning scope, identify, assess, remediate, then verify and repeat.
• A CVE ID names one flaw; databases add CVSS scores and affected products; CNAs assign the IDs.
• Prioritize with severity, exploitation evidence (KEV), exploitation likelihood (EPSS) and exposure, as CISA's BOD 26-04 does.
• Remediate by patching, rebuilding, reconfiguring, compensating, accepting with an expiry or retiring, and close nothing until a rescan confirms it.
• Scan only what you own, and treat scanners as part of your supply chain.

Sources

1. US Government Accountability Office. (2018). Data protection: Actions taken by Equifax and federal agencies in response to the 2017 breach (GAO-18-559). gao.gov
2. National Institute of Standards and Technology. (2017). CVE-2017-5638 detail. National Vulnerability Database; and CVE Program. CVE-2017-5638. nvd.nist.gov; cve.org
3. MITRE. (2024, October 22). CVE Program celebrates 25 years of impact in cybersecurity. mitre.org
4. FIRST. (2023). Common Vulnerability Scoring System version 4.0: Specification document; and (n.d.) Exploit Prediction Scoring System (EPSS). first.org (CVSS); first.org (EPSS)
5. Cybersecurity and Infrastructure Security Agency. (n.d.). Known Exploited Vulnerabilities Catalog; (2026, June 10) BOD 26-04: Prioritizing security updates based on risk; and (2021) BOD 22-01: Reducing the significant risk of known exploited vulnerabilities (revoked). cisa.gov (KEV); cisa.gov (BOD 26-04); cisa.gov (BOD 22-01)
6. Aqua Security. (2026, March 24). Trivy ecosystem supply chain was briefly compromised (GHSA-69fq-xp46-6x23). GitHub Advisory Database. github.com
7. Amazon Web Services. (n.d.). Penetration testing and What is Amazon Inspector? aws.amazon.com; docs.aws.amazon.com
8. Microsoft. (n.d.). Microsoft Cloud unified security testing rules of engagement and Vulnerability management for containers. microsoft.com; learn.microsoft.com
9. Google Cloud. (n.d.). Cloud security FAQ: penetration testing and Container scanning overview. support.google.com; docs.cloud.google.com
10. Anchore. (n.d.). Grype and Grype CLI reference. github.com; oss.anchore.com

One key, two breaches

In April 2018 the US Federal Trade Commission announced an expanded settlement with Uber. While the FTC was investigating a 2014 breach, the commission said, Uber had suffered another, strikingly similar one and failed to disclose it. In November 2016 Uber learned that intruders had again reached consumer data stored on its cloud provider's servers, this time using an access key that an Uber engineer had posted on a code-sharing website. With that key they downloaded unencrypted files holding more than 25 million names and email addresses, 22 million names and mobile phone numbers, and 600,000 names and driver's license numbers.

Debug that incident and at least three faults appear. A secret lived in code, where it could be copied. The key could read far more data than whoever used it needed. And the files themselves were readable by anyone holding a working key. None of those is exotic; each has a standard fix. This lesson works through a series of insecure configurations, finds the fault in each and applies the fix, covering CV0-004 objective 4.4: Zero Trust, benchmarks from CIS and the vendors, hardening, patching, encryption in transit and at rest, secrets management, API security, least privilege, container security and storage security.

Bug 1: "it is inside our network, so it is trusted"

A team lets services inside its virtual network call each other without authentication, because the network is private. That is the assumption Zero Trust rejects. NIST's Zero Trust Architecture, Special Publication 800-207, from August 2020, describes zero trust as moving defenses from static, network-based perimeters to users, assets and resources. It assumes no implicit trust is granted based solely on physical or network location or on who owns an asset. It treats authentication and authorization of both the user and the device as discrete functions performed before a session to a resource is established. Microsoft summarizes the strategy in three principles: verify explicitly, use least privilege access, and assume breach.

The fix is to authenticate and authorize every call, including service to service, for example with workload identities and mutual TLS. Segment the network so that one compromised service cannot reach everything, the failure Lesson 27 traced at Equifax, and grant each identity the least privilege it needs (Lesson 25). Zero Trust is a design approach, not a product, and it rests on the identity work of Lesson 25.

Bug 2: "we built it the way the installer suggested"

Default settings are chosen to make products easy to start, not hard to attack. Hardening removes what a system does not need and tightens what it keeps, and a benchmark says what "hardened" means so that it can be checked.

Typical hardening steps follow from any benchmark: remove unused packages and services, close every port that is not needed, disable direct root login and password-based SSH, start from minimal images, turn logging on, and patch on schedule as Lesson 24 described. Then scan continuously against the benchmark, because configuration drifts as people make changes, which is why infrastructure as code (Lesson 17) and policy checks belong together.

Bug 3: the key in the repository

The Uber key was a secret stored where code lives. OWASP's Secrets Management Cheat Sheet warns that secrets should never be hard-coded with Docker ENV or ARG instructions, because they leak with the container definition. It adds that environment variables are generally accessible to all processes and may end up in logs or system dumps. They are not recommended unless no other method is possible.

The fix is a secrets manager. AWS Secrets Manager describes replacing hard-coded credentials with a runtime call that retrieves them when needed, and rotating them through their life cycle. Azure Key Vault stores and controls access to tokens, passwords, certificates and API keys and also manages encryption keys. Google Cloud's Secret Manager stores API keys, passwords and certificates as versioned secrets. The workload reaches the manager with its own identity from Lesson 25, so no bootstrap password is needed. And a secret that has leaked is rotated, not merely deleted from the repository, because copies of the repository's history may already exist elsewhere.

Bug 4: "it is encrypted, so it is safe"

Encryption protects data in two places, and each layer stops different attacks.

Default encryption at rest is now universal. Amazon S3 applies server-side encryption with S3-managed keys as a base level for every bucket and has encrypted all new uploads automatically since 5 January 2023. Azure Storage encryption is enabled for all storage accounts and cannot be disabled. Google encrypts all customer content at rest without any action from the customer. Useful as that is, it would not have stopped the Uber download, because the storage service decrypts data for any caller it authorizes. A customer-managed key adds a second permission an attacker must also obtain, and it makes cryptographic erase possible (Lesson 24). In transit, NIST's TLS guidance, SP 800-52 Revision 2, requires US government servers and clients to support TLS 1.2 and to support TLS 1.3 by 1 January 2024, a sensible floor for anyone.

Bug 5: the API that trusts its callers

Any logged-in user of an invoicing API can read anyone's invoice by changing the number in the URL. The OWASP API Security Top 10 of 2023 puts this flaw first: API1, broken object level authorization. The API authenticated the caller but never checked that this caller may see this object. Its neighbours on the list include broken authentication (API2) and unrestricted resource consumption (API4), the missing rate limits that let one client exhaust a service or its budget. The fixes are to authenticate every call, authorize every object, validate input, enforce quotas and rate limits, often at an API gateway, use TLS, and log access.

In short: trust nothing because of where it sits, give every identity only what it needs, keep secrets in a manager, encrypt in transit and at rest with keys you control where it matters, and check every API call for both who is calling and what they may touch.

Bug 6: the privileged container

A monitoring container keeps hitting permission errors, so someone adds --privileged. Docker's reference explains what that does: it enables all Linux kernel capabilities, disables the default seccomp and AppArmor profiles and the SELinux process label, grants access to all host devices and makes /sys and the cgroup mounts writable. In Docker's words, the container can then do almost everything that the host can do; it is not a securely sandboxed process and can get a root shell on the host. NIST's container security guide, SP 800-190, makes the same point and adds that containers should run with their root file systems in read-only mode.

Kubernetes enforces these choices with its Pod Security Standards. The Privileged profile is unrestricted; Baseline prevents known privilege escalations while allowing a default pod; Restricted follows current hardening practice. Under Restricted, containers must run as non-root, must not allow privilege escalation, must use a seccomp profile and must drop all capabilities, adding back at most NET_BIND_SERVICE. A namespace label, pod-security.kubernetes.io/enforce=restricted, makes the cluster reject pods that break those rules.

Bug 7: the bucket and the share

For object storage, AWS says new buckets, access points and objects do not allow public access by default, but users can change bucket policies and object permissions to allow it. S3 Block Public Access settings override such policies, so keep them on at the account level and turn them off only for a bucket that truly serves the public. To share one private object for a limited time, use a presigned URL, which grants time-limited permission to download it, rather than opening the bucket; Azure and Google Cloud offer equivalents in shared access signatures and signed URLs. Add versioning and object locks where deletion or ransomware is a risk (Lessons 23 and 26), and access logging.

For file storage, shared over NFS or SMB, the same principles apply in a different shape. Allow network access only from the subnets or private endpoints that need the share, and authenticate users with the directory rather than shared passwords. Set share and file permissions to least privilege, and require encryption in transit where the protocol version supports it.

Common misconceptions

• "Our data is encrypted at rest, so a leaked key cannot expose it." The storage service decrypts for any authorized caller; only separate key permissions change that.
• "A private repository is a safe place for keys." Repositories are cloned, forked and shared; secrets belong in a secrets manager.
• "Root in a container is harmless because containers are isolated." Isolation weakens with every privilege granted, and --privileged removes most of it.
• "Zero Trust is a product." It is an approach: verify explicitly, grant least privilege and assume breach.

Where this leaves us

• Zero Trust removes trust based on network location: authenticate and authorize every session, segment, and grant least privilege.
• Benchmarks such as CIS, AWS Foundational Security Best Practices and the Microsoft cloud security benchmark define hardening so it can be measured and kept, alongside routine patching.
• Secrets belong in a secrets manager, reached with the workload's identity and rotated after any exposure.
• Encrypt in transit with TLS 1.2 or later and at rest, using customer-managed keys where a second gate or cryptographic erase matters.
• Secure APIs object by object, run containers unprivileged and read-only, and keep storage private by default, sharing through time-limited URLs.

Sources

1. Federal Trade Commission. (2018, April 12). Uber agrees to expanded settlement with FTC related to privacy, security claims. ftc.gov
2. Rose, S., Borchert, O., Mitchell, S., and Connelly, S. (2020). Zero trust architecture (NIST Special Publication 800-207). National Institute of Standards and Technology. csrc.nist.gov
3. Center for Internet Security. (n.d.). CIS Benchmarks. cisecurity.org
4. Amazon Web Services. (n.d.). AWS Foundational Security Best Practices standard, What is AWS Secrets Manager?, Protecting data with server-side encryption, Blocking public access to your Amazon S3 storage and Sharing objects with presigned URLs. docs.aws.amazon.com (FSBP); docs.aws.amazon.com (Secrets Manager); docs.aws.amazon.com (encryption); docs.aws.amazon.com (public access); docs.aws.amazon.com (presigned URLs)
5. Microsoft. (n.d.). Overview of the Microsoft cloud security benchmark v2, About Azure Key Vault and Azure Storage encryption for data at rest. Microsoft Learn. learn.microsoft.com (benchmark); learn.microsoft.com (Key Vault); learn.microsoft.com (storage encryption)
6. Google Cloud. (n.d.). Default encryption at rest and Secret Manager overview. docs.cloud.google.com (encryption); docs.cloud.google.com (Secret Manager)
7. OWASP Foundation. (n.d.). Secrets management cheat sheet and OWASP API Security Top 10 2023. cheatsheetseries.owasp.org; owasp.org
8. McKay, K., and Cooper, D. (2019). Guidelines for the selection, configuration, and use of Transport Layer Security (TLS) implementations (NIST Special Publication 800-52 Revision 2). National Institute of Standards and Technology. csrc.nist.gov
9. Souppaya, M., Morello, J., and Scarfone, K. (2017). Application container security guide (NIST Special Publication 800-190). National Institute of Standards and Technology. csrc.nist.gov
10. Docker. (n.d.). docker container run reference. docs.docker.com; The Kubernetes Authors. (n.d.). Pod Security Standards and Pod Security Admission. kubernetes.io (standards); kubernetes.io (admission)

398 million requests a second

In August 2023 Google stopped a distributed denial-of-service attack that peaked above 398 million requests per second; the largest attack recorded the year before had peaked at 46 million. Google's account, published on 10 October 2023, put the scale in perspective: the two-minute attack generated more requests than Wikipedia reported article views for the whole of September 2023. The attackers used a new technique called HTTP/2 Rapid Reset, which abuses the protocol's stream multiplexing, and the weakness was tracked as CVE-2023-44487 with a CVSS score of 7.5. CISA added it to its known exploited catalog the same day the details were published.

No single firewall rule could have stopped it, because every request was valid HTTP/2. Google absorbed it at the edge of its network with capacity, updated proxies and denial-of-service defenses. It noted that Cloud customers behind its Application Load Balancer and Cloud Armor were protected the same way, because those services run on the same infrastructure. Defense came from layers, each seeing something the others could not. This lesson compares those layers, the security controls in CV0-004 objective 4.5: firewalls in the form of network access control lists, network security groups and web application firewalls; intrusion detection and prevention; DDoS protection; data loss prevention; endpoint protection; and IAM policies.

The controls at a glance

Network filters compared: stateless and stateful

The difference that matters most is memory. AWS describes network ACLs as stateless: information about previously sent or received traffic is not saved, so if a rule allows specific inbound traffic to a subnet, the responses are not automatically allowed. Rules are evaluated in order, starting with the lowest number. A web server behind a network ACL therefore needs an outbound rule for responses to clients' ephemeral ports, as well as the inbound rule for HTTPS. Security groups are stateful: if a request is allowed, its response is allowed regardless of the rules in the other direction. AWS security groups accept allow rules but not deny rules, and a new security group has no inbound rules, so it admits no inbound traffic until you add some.

Microsoft notes that the flow record is what lets an NSG be stateful: allow outbound traffic to port 80 and you need no inbound rule for the responses. Google describes VPC firewall rules as defined at the network level but enforced per instance, protecting instances even before they have started. A common pattern uses both kinds: a network ACL for coarse subnet-wide denies, such as blocking an abusive address range, and security groups for the precise allow rules of each tier.

The upshot: stateless filters need rules in both directions and can deny; stateful filters remember connections and, at AWS, only allow. Pick the layer by what you need to express, then remember that neither looks inside the traffic it admits.

WAF, IDS and IPS compared

NIST's guide to intrusion detection and prevention systems, SP 800-94, defines intrusion detection as monitoring events in a system or network and analyzing them for signs of possible incidents. It defines intrusion prevention as performing intrusion detection and attempting to stop the incidents detected. Placement decides which is possible. An inline sensor sits in the traffic path, like a firewall. A passive sensor watches a copy, and because it monitors a copy, NIST notes, it typically has no reliable way to stop traffic reaching its destination. Prevention generally needs inline placement.

AWS describes its WAF as monitoring the HTTP(S) requests forwarded to protected resources. Microsoft names SQL injection and cross-site scripting as among the most common attacks its WAF stops, and Google says Cloud Armor's preconfigured WAF rules are based on the OWASP ModSecurity Core Rule Set. A WAF is also the usual virtual patch while a fix is prepared (Lesson 27). Every one of these controls needs to see plaintext. A WAF at the load balancer can inspect requests after TLS terminates there. A sensor shown only encrypted traffic is as blind as Equifax's inspection tool was with its expired certificate.

DDoS protection

Floods come in two broad kinds: volumetric attacks that fill network links at layers 3 and 4, and application-layer attacks such as Rapid Reset that send valid-looking requests faster than servers can answer. The providers absorb the first kind at their edge by default. AWS says all its customers benefit from the automatic protection of AWS Shield Standard at no additional charge. Azure applies default infrastructure-level protection against common network-layer attacks and sells DDoS Network Protection, tuned to resources in a virtual network, and a pay-per-IP DDoS IP Protection tier. Google Cloud Armor protects deployments from DDoS attacks as well as application attacks.

The design rules follow. Put public services behind the provider's load balancers and CDN rather than exposing origin servers directly, and add WAF rate limits for application floods. Treat autoscaling as a way to keep serving during a surge, not as a defense. A service that scales to meet a flood also scales its bill (Lessons 14 and 22).

DLP, endpoint protection and IAM policies

Data loss prevention watches the data rather than the network. Microsoft describes DLP as helping prevent users from inappropriately sharing sensitive data with people who should not have it, through policies that identify, monitor and automatically protect sensitive information across applications, devices and web traffic. The discovery services from Lesson 26 find where sensitive data lives; DLP policies act when it moves, warning on or blocking a spreadsheet of card numbers bound for a personal email account.

Endpoint protection watches the hosts. On virtual machines that means anti-malware and endpoint detection and response agents. For containers it means runtime security tools such as Falco. Its project describes Falco as providing runtime security across hosts, containers, Kubernetes and cloud environments by applying rules to Linux kernel events. A container that suddenly starts a shell or writes to /etc is exactly what such rules catch.

IAM policies are controls too, and at organization scale they become guardrails. AWS service control policies never grant permissions; they set the maximum permissions available to users and roles across an organization's accounts. Azure Policy enforces organizational standards and assesses compliance at scale, and Google's Organization Policy Service gives centralized, programmatic control through constraints across the resource hierarchy. Guardrails can forbid disabling audit logs, creating public buckets or deploying outside approved regions (Lesson 26), whatever an individual account's administrator attempts.

Common misconceptions

• "Security groups and network ACLs are interchangeable." One is stateful and, at AWS, allow-only; the other is stateless, can deny and needs rules for return traffic.
• "A WAF protects every port." It understands HTTP; other protocols need network firewalls and IPS rules.
• "An IDS blocks attacks." It detects and alerts. Blocking needs an inline IPS or another control acting on the alert.
• "Autoscaling is DDoS protection." It serves the flood and sends you the bill; filtering happens at the edge and in rate limits.

Pulling it together

• Network ACLs filter subnets statelessly and can deny; security groups and NSGs filter resources statefully; Google's VPC firewall rules are stateful and enforced per instance.
• A WAF reads HTTP; an IDS watches a copy of traffic and alerts; an IPS sits inline and blocks.
• Providers absorb common floods by default; application floods need edge services, rate limits and budget-aware scaling.
• DLP protects data in motion, endpoint protection watches hosts and containers at runtime, and organization-wide IAM policies set guardrails no single administrator can cross.

Sources

1. Google Cloud. (2023, October 10). Google mitigated the largest DDoS attack to date, peaking above 398 million rps. Google Cloud Blog. cloud.google.com
2. Scarfone, K., and Mell, P. (2007). Guide to intrusion detection and prevention systems (IDPS) (NIST Special Publication 800-94). National Institute of Standards and Technology. csrc.nist.gov
3. Amazon Web Services. (n.d.). Control subnet traffic with network access control lists, Control traffic to your AWS resources using security groups and Security group rules. docs.aws.amazon.com (network ACLs); docs.aws.amazon.com (security groups); docs.aws.amazon.com (rules)
4. Amazon Web Services. (n.d.). What is AWS WAF?, What is AWS Network Firewall?, AWS Shield Standard overview and Service control policies. docs.aws.amazon.com (WAF); docs.aws.amazon.com (Network Firewall); docs.aws.amazon.com (Shield); docs.aws.amazon.com (SCPs)
5. Microsoft. (n.d.). Network security groups, What is Azure Web Application Firewall?, Azure Firewall Premium features, Azure DDoS Protection overview, Overview of Azure Policy and Learn about data loss prevention. Microsoft Learn. learn.microsoft.com (NSG); learn.microsoft.com (WAF); learn.microsoft.com (Firewall Premium); learn.microsoft.com (DDoS); learn.microsoft.com (Policy); learn.microsoft.com (DLP)
6. Google Cloud. (n.d.). VPC firewall rules, Cloud Armor overview and Introduction to the Organization Policy Service. docs.cloud.google.com (firewall); docs.cloud.google.com (Cloud Armor); docs.cloud.google.com (Organization Policy)
7. The Falco Project. (n.d.). Falco. falco.org; OWASP CRS Project. (n.d.). ModSecurity Core Rule Set Docker images. github.com

Found by a stranger

On 17 July 2019 an external security researcher sent Capital One a report through the bank's Responsible Disclosure Program. It described a configuration vulnerability. The bank's own investigation followed, and on 19 July 2019 it determined that an outside individual had gained unauthorized access to personal information about its credit card customers and applicants. The bank later put the number affected at about 100 million people in the United States and about 6 million in Canada. The individual was caught by the FBI, and the government said it believed the data had been recovered.

The detail that matters for this lesson is the order of events. The intrusion was not spotted by the bank's monitoring; it was reported by someone outside. Every attack leaves evidence in a cloud account: API calls, network flows, processes, bills. The question is whether anyone is watching the right evidence, knows what normal looks like, and notices when it changes. This lesson follows an attack through the evidence it leaves and covers CV0-004 objective 4.6: event monitoring, deviation from the baseline, unnecessary open ports, and the attack types the objective lists, from vulnerability exploitation, phishing and ransomware to DDoS, cryptojacking, zombie instances and metadata abuse.

Where the evidence lands

Nobody reads all of that by hand. Managed detectors read it for you. Amazon GuardDuty is a threat detection service that continuously monitors AWS data sources and logs, including CloudTrail management events, VPC flow logs and DNS logs, using threat intelligence and machine learning models. Microsoft Sentinel is a cloud-native SIEM, a security information and event management system, for threat detection, investigation and response across environments. Google's Event Threat Detection, part of Security Command Center, monitors an organization or project and identifies threats in near-real time. Event monitoring means collecting these sources centrally, protecting them, and routing the findings to someone who acts.

Knowing normal

A deviation needs a baseline to deviate from. Amazon CloudWatch's anomaly detection, for example, applies statistical and machine learning algorithms to metrics to determine normal baselines and produce a band of expected values. An alarm can then fire when a metric leaves the band rather than when it crosses a fixed line. AWS Cost Anomaly Detection does the same for spending. Some deviations are worth learning by heart.

One night in an account

Follow an imaginary but typical attack through those sources.

Midnight: the knock on open ports. Automated scanners probe every public address on the internet, so any port left open is found quickly. CISA's ransomware guide tells organizations not to expose services such as the Remote Desktop Protocol on the web and to disable ports and protocols that are not used for business purposes. An unnecessary open port is not an attack yet, but it is an invitation. A regular check of what listens on each public address is one of the cheapest controls there is.

01:00: the way in. CISA groups the common initial access vectors of ransomware and data extortion actors under internet-facing vulnerabilities and misconfigurations, compromised credentials, and phishing. The first two are the objective's vulnerability exploitation through outdated software, as at Equifax (Lesson 27), and through human error, such as a misconfigured firewall or a public bucket. Phishing is social engineering: a convincing message that persuades someone to hand over credentials or codes. That is why CISA recommends phishing-resistant MFA for all services and training staff to recognize and report suspicious messages.

02:00: the metadata service. Every cloud instance can ask a local metadata service about itself, at the link-local address 169.254.169.254, and on AWS that service also hands out the temporary credentials of the instance's role. A web application can sometimes be tricked into making requests on an attacker's behalf, through server-side request forgery, an open reverse proxy or a misconfigured WAF. The attacker can then ask the metadata service for those credentials.

AWS's 2019 answer was IMDSv2. With IMDSv2, every request needs a session token obtained with an HTTP PUT request, which the vast majority of WAFs do not pass through. The service also refuses tokens to requests carrying an X-Forwarded-For header, the mark of a proxy. Azure's metadata service similarly rejects requests that lack a Metadata: true header or carry X-Forwarded-For, and Google's requires the Metadata-Flavor: Google header. Require IMDSv2 on every AWS instance. If credentials leak anyway, GuardDuty raises a high-severity finding, UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS, when credentials created exclusively for an instance are used from an external IP address.

03:00: the miners arrive. Cryptojacking turns someone else's compute into cryptocurrency. Microsoft's threat intelligence team describes attackers who use compromised credentials to create large amounts of compute in a tenant, install mining software and join mining pools. It reports that targeted organizations incurred more than 300,000 US dollars in compute fees. The evidence is loud if anyone listens: new virtual machines in unused regions, quota increases, CPU pinned at 100 percent, spend anomalies and network findings. GuardDuty's CryptoCurrency:EC2/BitcoinTool.B, for example, reports an instance querying an IP address associated with cryptocurrency-related activity.

04:00: ransomware. Malware that encrypts data and demands payment now usually tries to destroy recovery first. CISA notes that many ransomware variants attempt to find and delete or encrypt accessible backups, which is why it advises offline, encrypted backups tested regularly, the immutable, separate-account copies of Lesson 23. In the logs, ransomware appears as mass rewrites and deletions and as attempts to delete backups, snapshots or retention settings.

Daylight: the flood. A DDoS attack, the subject of Lesson 29, is the loudest attack of all: request rates and error rates leave their baseline together. It is sometimes also a distraction from quieter activity elsewhere, so a flood is a reason to look at the audit trail too.

Worth holding on to: almost every step above changed something measurable: a port answered, an API was called from a strange place, a region woke up, a CPU stayed busy, a bill jumped. Monitoring is the habit of knowing those numbers well enough to notice.

Zombie instances

Some of the most dangerous resources in an account are the ones nobody remembers. AWS's migration guidance calls applications whose average CPU and memory use is below 5 percent zombie applications. It calls those between 5 and 20 percent over 90 days idle applications, and both are candidates for retirement. A zombie instance is the same thing seen from security's side: a server still running, still billing and still reachable, but owned by nobody, patched by nobody and watched by nobody. It may still hold credentials, open ports and old software. It is exactly the kind of machine an attacker can take over unnoticed and turn into a zombie in the older security sense, a compromised machine that works for someone else. The fixes are those of earlier lessons: owner tags enforced by policy (Lesson 14), a regularly reconciled inventory, alerts on long-idle resources, and the decommissioning checklist of Lesson 24.

Common misconceptions

• "We would notice an intrusion." Without collected, protected and reviewed logs, many organizations learn of intrusions from outsiders, as Capital One did.
• "Cryptojacking is only a cost problem." It proves an attacker holds working credentials, which can be used for theft as well as mining.
• "The metadata service is only reachable from the instance, so it is safe." Anything that can make the instance send requests, such as a proxy or SSRF flaw, can reach it; IMDSv2 and header requirements close that path.
• "An idle server is harmless." Idle often means unpatched and unwatched, which makes it an ideal foothold.

Summing up

• Event monitoring gathers audit trails, flow logs, DNS logs, host logs, metrics and billing, and managed detectors such as GuardDuty, Sentinel and Event Threat Detection turn them into findings.
• Baselines make deviations visible: CPU at night, new regions, unusual egress, outside use of instance credentials, mass writes, sudden spend.
• Close unnecessary open ports; most attacks begin with exposed services, unpatched software, misconfiguration, stolen credentials or phishing.
• Require IMDSv2 and the providers' metadata headers, and alert on instance credentials used from outside.
• Cryptojacking, ransomware and DDoS each have loud signatures; zombie instances are quiet risks to find and retire.

Sources

1. Capital One. (n.d.). Information on the Capital One cyber incident. capitalone.com
2. MacCárthaigh, C. (2019, November 19). Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service. AWS Security Blog. aws.amazon.com
3. Microsoft Threat Intelligence. (2023, July 25). Cryptojacking: Understanding and defending against cloud compute resource abuse. Microsoft Security Blog. microsoft.com
4. Cybersecurity and Infrastructure Security Agency. (n.d.). #StopRansomware guide. cisa.gov
5. Amazon Web Services. (n.d.). What is Amazon GuardDuty?, GuardDuty EC2 finding types, GuardDuty IAM finding types, Using CloudWatch anomaly detection and Detecting unusual spend with AWS Cost Anomaly Detection. docs.aws.amazon.com (GuardDuty); docs.aws.amazon.com (EC2 findings); docs.aws.amazon.com (IAM findings); docs.aws.amazon.com (anomaly detection); docs.aws.amazon.com (cost anomalies)
6. Amazon Web Services. (n.d.). Migration strategies. AWS Prescriptive Guidance. docs.aws.amazon.com
7. Microsoft. (n.d.). Azure Instance Metadata Service and What is Microsoft Sentinel? Microsoft Learn. learn.microsoft.com (IMDS); learn.microsoft.com (Sentinel)
8. Google Cloud. (n.d.). Access VM metadata and Event Threat Detection overview. docs.cloud.google.com (metadata); docs.cloud.google.com (Event Threat Detection)

Seven servers out of eight

In late July 2012 Knight Capital, a large US market maker, deployed new code to its automated order router for a new New York Stock Exchange program. According to the SEC's order, one of its technicians did not copy the new code to one of the eight servers. No second technician reviewed the deployment, and the router had no written code deployment procedures. The eighth server still held old code, including a function left unused in the router since 2005 that the new orders now triggered. In the first 45 minutes after the market opened on 1 August, the router sent more than 4 million orders while trying to fill 212 customer orders. It traded more than 397 million shares and left Knight with a loss of more than 460 million dollars. Before the open, an internal system had sent 97 automated emails that referred to the router and identified an error; nobody acted on them. The SEC fined the firm 12 million dollars in 2013.

Every failure in that sequence has a DevOps answer. Keep all code, including dead code, under version control, and have every change reviewed. Build one artifact and deploy it identically to every server by machine, not by hand; test it automatically; and make alerts reach someone who acts. This lesson builds those habits step by step. It covers CV0-004 objective 5.1, source control concepts, and 5.2, CI/CD pipelines, and introduces three of the tools objective 5.4 names: Git, GitHub Actions and Jenkins.

Step 1: put everything under version control

Git is the version control system almost every team uses. The Pro Git book explains that Git thinks of its data as a series of snapshots of a miniature file system, and that files sit in three states: modified, staged and committed. Once a snapshot is committed it is very difficult to lose, especially if it is regularly pushed to another repository. The objective's terms map onto everyday Git work.

Version everything that defines the system: application code, infrastructure as code (Lesson 17), configuration as code (Lesson 18), pipeline definitions and documentation. Never version secrets (Lesson 28). A repository that holds all of this can rebuild the system and explain every change to it.

Step 2: review every change before it reaches main

Git itself has no pull requests; hosting platforms add them. GitHub describes pull requests as proposals to merge code changes into a project, letting people discuss and review changes before merging them. The usual flow keeps main always releasable: create a short-lived branch, commit and push, open a pull request, let reviewers and automated checks examine it, then merge and delete the branch.

Rules turn the habit into a guarantee. GitHub's branch protection rules can require pull request reviews before merging and require status checks, such as a passing pipeline, before a change lands; they can also block force pushes and deletion. GitHub Free offers protected branches on public repositories, and paid plans add them for private ones. Knight Capital's missing second technician is exactly the gap a required review closes.

Step 3: integrate continuously

Martin Fowler's widely cited definition describes continuous integration as each member of a team merging changes into the shared codebase at least daily. Each integration is verified by an automated build, including tests, to detect errors as quickly as possible. A pipeline makes that automatic.

The last steps come in two flavours. AWS describes continuous delivery as automatically preparing every change for a release to production. It draws the line plainly: the difference between continuous delivery and continuous deployment is the presence of a manual approval to update production. With continuous deployment, production updates happen automatically without explicit approval.

Step 4: build once and keep the artifacts

An artifact is what the pipeline produces and deploys. Building it once and promoting that same artifact from test to staging to production is what guarantees that production runs exactly what was tested, the opposite of copying files to servers by hand.

Repositories are public or private. Public ones, such as Docker Hub's public images or a project's open-source packages, serve anyone; private ones, such as a company's own container registry, hold internal artifacts behind access control and scanning. Pull public artifacts by pinned version or digest, and publish internal ones only to private repositories.

Step 5: choose the pipeline engine

Jenkins describes its Pipeline as a suite of plugins for implementing continuous delivery pipelines, defined in a Jenkinsfile that is committed to source control, which it calls pipeline as code. AWS calls CodePipeline a continuous delivery service that models, visualizes and automates release stages. Microsoft describes Azure Pipelines as combining continuous integration, continuous testing and continuous delivery. Google's Cloud Build executes builds on Google Cloud.

Pipelines hold the keys to production, so they are prime targets. In March 2025 the popular third-party GitHub Action tj-actions/changed-files was compromised. CISA reported that the malicious version exposed secrets such as access keys, personal access tokens and private keys in workflow logs, and told affected users to rotate every exposed secret. A year later the Trivy action suffered the same fate (Lesson 27). GitHub's security guidance is to pin actions to a full-length commit SHA, because a bad actor would then need a SHA-1 collision to slip in a backdoor. A version tag, by contrast, can simply be moved. Pipelines should also reach clouds with short-lived OpenID Connect credentials rather than stored keys (Lesson 25).

The optional GitHub step in this lesson's lab needs a GitHub account. GitHub's terms require users to be 13 or older, or older where a country sets a higher minimum age, and a free account needs no payment details.

So what?: Knight Capital lost 460 million dollars to a manual, unreviewed, partial deployment and ignored alerts. Version control, required review, a pipeline that builds once, tests and deploys the same artifact everywhere, and alerts that reach a person are the cheap insurance against that kind of morning.

Common misconceptions

• "Git and GitHub are the same thing." Git is the version control system; GitHub hosts Git repositories and adds pull requests, reviews and Actions.
• "Having a build server means we do CI." CI means integrating at least daily, with every integration verified by an automated build and tests.
• "Continuous delivery and continuous deployment are synonyms." Delivery keeps a manual approval before production; deployment does not.
• "Pinning to a version tag is as safe as pinning to a SHA." Tags can be repointed, as the tj-actions and Trivy compromises showed.

Looking back

• Commits record snapshots, pushes share them, branches isolate work, merges combine it and tags name versions.
• Pull requests and code review, enforced by branch protection, keep main releasable.
• A CI/CD pipeline automates build, test, security, packaging and deployment, and stops on the first failure.
• Build one artifact, whether a VM image, container image, package or flat file, store it in a public or private repository, and promote it unchanged.
• GitHub Actions, Jenkins and provider services all run pipelines; pin third-party actions to commit SHAs and use short-lived credentials.

Sources

1. US Securities and Exchange Commission. (2013, October 16). SEC charges Knight Capital with violations of market access rule (Press release 2013-222), and In the matter of Knight Capital Americas LLC (Release No. 34-70694). sec.gov (press release); sec.gov (order)
2. Chacon, S., and Straub, B. (n.d.). Pro Git (2nd ed.), What is Git? git-scm.com
3. GitHub. (n.d.). Pull requests, About protected branches, Secure use reference and GitHub Terms of Service. GitHub Docs. docs.github.com (pull requests); docs.github.com (protected branches); docs.github.com (secure use); docs.github.com (terms)
4. Fowler, M. (2024). Continuous integration. martinfowler.com
5. Amazon Web Services. (n.d.). What is continuous delivery? and What is AWS CodePipeline? aws.amazon.com; docs.aws.amazon.com
6. Cybersecurity and Infrastructure Security Agency. (2025, March 18). Supply chain compromise of third-party tj-actions/changed-files (CVE-2025-30066) and reviewdog/action-setup@v1 (CVE-2025-30154). cisa.gov
7. Jenkins project. (n.d.). Jenkins and Pipeline. jenkins.io; jenkins.io (Pipeline)
8. Microsoft. (n.d.). What is Azure Pipelines? Microsoft Learn. learn.microsoft.com; Google Cloud. (n.d.). Cloud Build overview. docs.cloud.google.com
9. RPM project. (n.d.). RPM Package Manager. rpm.org

Two or three calls for one view

On 14 September 2016 GitHub announced that its API would also be available through GraphQL. Its REST API, the company explained, was responsible for over 60 percent of the requests made to its database tier. Part of the reason was that hypermedia navigation makes a client go back to the server again and again for what it needs. Integrators complained that assembling a complete view of one resource sometimes took two or three separate calls, and that responses simultaneously sent too much data and left out data they needed. GitHub did not abandon REST; it added a second way in, chosen for a different trade-off.

Every integration makes trade-offs like that one: request and response or events, a fixed resource shape or a query, a short call or a connection held open. This lesson compares the integration styles in CV0-004 objective 5.3, web services built on REST, SOAP and RPC, web sockets, GraphQL and event-driven architectures. It then compares, side by side, the nine tools that objective 5.4 names and earlier lessons have used.

Web services compared: REST, SOAP and RPC

REST is an architectural style, not a protocol. Roy Fielding defined it in his 2000 dissertation through a set of constraints. Client and server are separated, and communication is stateless, so that each request carries all the information needed to understand it and session state stays with the client. Responses are labelled cacheable or not, the interface is uniform, the system is layered and, optionally, code can be sent on demand. Statelessness is what lets a load balancer send any request to any server (Lessons 7 and 22).

SOAP, in the words of its W3C Recommendation, is a lightweight protocol intended for exchanging structured information in a decentralized, distributed environment, and it defines an extensible messaging framework using XML technologies. It remains common where formal contracts and established XML standards matter, such as older finance and government integrations. RPC styles put the emphasis on calls rather than resources. The gRPC documentation describes a client calling a method on a server on a different machine as if it were a local object, with Protocol Buffers by default.

Beyond request and response: web sockets and GraphQL

Some applications need the server to speak first. RFC 6455 describes the WebSocket Protocol as enabling two-way communication between a client and a remote host, using an opening handshake followed by basic message framing, layered over TCP. A live dashboard that polls a REST endpoint every 5 seconds from 1,000 browsers makes 720,000 requests an hour, mostly to learn that nothing has changed. Over web sockets each browser holds one connection and the server pushes changes when they happen. The price is long-lived connections that load balancers, scaling and deployments must all handle.

GraphQL attacks GitHub's problem. Its foundation describes it as a query language for your API and a server-side runtime for executing queries using a type system you define for your data. A client sends one query, usually to a single endpoint, naming exactly the fields it wants, and receives exactly those. The costs move elsewhere: caching is harder than for plain GET requests, and an unrestricted query can be expensive, so servers limit query depth and cost, the resource-consumption risk from Lesson 28.

Event-driven architecture

AWS describes an event-driven architecture as one that uses events to trigger and communicate between decoupled services. An event is a change in state, such as an item placed in a shopping cart. There are three components: producers publish events to a router, and the router filters them and pushes them to consumers. Because producers and consumers are decoupled, they can be scaled, updated and deployed independently. When an order is placed, the order service publishes one event. Billing, shipping and email each subscribe, and a fourth consumer can be added later without touching the order service, the fan-out pattern from Lesson 11.

Every provider offers routers and queues. Amazon EventBridge is a serverless service that uses events to connect application components, and Azure Event Grid is a fully managed publish-subscribe service. Google Cloud Pub/Sub is an asynchronous messaging service that decouples the services producing messages from those processing them. Delivery guarantees differ and must be read, not assumed. Redis's documentation, for instance, says its pub/sub channels deliver at most once, so a consumer that is offline simply misses the message, while durable queues and streams keep events until consumers read them. Events also make a request's path harder to follow, which is why distributed tracing (Lesson 21) matters more in event-driven systems.

The core of it: choose request and response when the caller needs an answer now, and web sockets when the server must push. Use GraphQL when clients need flexible slices of data, RPC for efficient internal calls and SOAP for formal XML partners. Use events when many services must react to one change without knowing about each other.

The nine DevOps tools compared

Together they form one loop. A change is committed to Git, GitHub Actions or Jenkins runs the pipeline, and Docker builds the image. Terraform provisions the network, cluster and databases, Ansible configures any virtual machines and Kubernetes runs the containers. The ELK stack gathers their logs, Grafana shows their metrics, and what the team learns becomes the next commit.

Four pairs are easy to confuse. Terraform and Ansible overlap, but Terraform declares the infrastructure that should exist, while Ansible configures what runs inside it. Docker builds and runs containers on one machine; Kubernetes schedules and heals them across many. Jenkins is a server you operate; GitHub Actions is a service attached to your repositories. The ELK stack is a log pipeline with its own store and interface; Grafana is a visualization layer that can sit on top of Elasticsearch, Prometheus and many other sources at once.

Common misconceptions

• "REST is a protocol." It is an architectural style; HTTP is the protocol REST APIs usually use.
• "GraphQL replaces REST." It trades simpler caching for flexible queries; GitHub kept both.
• "Event-driven means instant." It means decoupled and usually asynchronous; delivery can be delayed, repeated or, on some channels, lost.
• "Kubernetes replaced Docker." They work at different layers: one builds and runs containers, the other orchestrates them.

What to remember

• REST uses resources, HTTP methods and stateless requests; SOAP exchanges XML messages under a formal framework; RPC calls remote functions, as gRPC does with Protocol Buffers.
• Web sockets keep a two-way connection open so servers can push; GraphQL lets clients ask for exactly the fields they need.
• Event-driven architectures connect producers and consumers through routers, decoupling them; read each service's delivery guarantees.
• Git, GitHub Actions, Jenkins, Terraform, Ansible, Docker, Kubernetes, the ELK stack and Grafana each own one part of the delivery loop.

Sources

1. GitHub. (2016, September 14). The GitHub GraphQL API. The GitHub Blog. github.blog
2. Fielding, R. T. (2000). Architectural styles and the design of network-based software architectures, chapter 5: Representational State Transfer (REST) [Doctoral dissertation, University of California, Irvine]. ics.uci.edu
3. World Wide Web Consortium. (2007). SOAP version 1.2 part 1: Messaging framework (2nd ed.). W3C Recommendation. w3.org
4. gRPC Authors. (n.d.). Introduction to gRPC. grpc.io
5. Fette, I., and Melnikov, A. (2011). The WebSocket protocol (RFC 6455). Internet Engineering Task Force. rfc-editor.org
6. GraphQL Foundation. (n.d.). Introduction to GraphQL. graphql.org
7. Amazon Web Services. (n.d.). What is an event-driven architecture? and What is Amazon EventBridge?; Microsoft. (n.d.). What is Azure Event Grid?; Google Cloud. (n.d.). What is Pub/Sub?; Redis. (n.d.). Redis Pub/Sub. aws.amazon.com; docs.aws.amazon.com; learn.microsoft.com; docs.cloud.google.com; redis.io
8. Tool documentation: Git (git-scm.com); GitHub Actions (docs.github.com); Jenkins (jenkins.io); Terraform (developer.hashicorp.com); Ansible (docs.ansible.com); Docker (docs.docker.com); Kubernetes (kubernetes.io); the Elastic Stack (elastic.co); Grafana (grafana.com)

Seventy-eight minutes

At 04:09 UTC on Friday 19 July 2024, CrowdStrike released a routine content configuration update for its Falcon sensor on Windows. According to the company's preliminary post-incident review, a bug in its Content Validator let one of two new template instances pass validation despite containing problematic content data. When sensors loaded it, the content caused an out-of-bounds memory read that could not be handled gracefully, and Windows crashed. The update was reverted at 05:27 UTC, 78 minutes later, but every Windows host that had received it in that window was already down. Microsoft estimated that 8.5 million Windows devices were affected, less than one percent of all Windows machines, and the disruption reached airlines, hospitals and banks. Among CrowdStrike's announced fixes were more validation and a staggered deployment strategy for such updates, starting with a canary deployment.

A deployment is a change, and changes are the commonest cause of failures. This lesson debugs a series of broken deployments, covering CV0-004 objective 6.1: incompatibility; misconfigurations of resource allocation, permissions, oversubscription and sizing; outdated component definitions; deprecated functionality; full and partial outages; resource limits from API throttling and service quotas; and regional service availability.

A method before the bugs

Use the troubleshooting method from Network+ Certification Prep (cs-netplus, CS 215): identify the problem, establish a theory, test it, plan the fix and its effects, implement or escalate, verify and prevent, and document throughout. Two questions start every deployment investigation. First, what changed: a release, a configuration, a quota, a provider retirement? Second, how wide is the failure: everything, one region, one zone, one percent of requests? The answers sort most problems into the categories below before a single log line is read.

Bug 1: Pending pods and killed containers

A deployment rolls out and its pods sit in Pending; kubectl describe shows the scheduler's event, "Insufficient cpu". The pods request more CPU than any node has free, a resource allocation fault, so the fix is to lower the requests to what the application really needs or add capacity (Lesson 22). A different deployment starts but restarts endlessly with exit code 137 and the reason OOMKilled: its memory limit is smaller than its real usage, a sizing fault fixed by measuring and right-sizing (Lesson 19), not by removing limits altogether.

Bug 2: AccessDenied in the pipeline

A pipeline's infrastructure step fails with AccessDenied while creating a load balancer. The error names the action and the resource, and the audit trail (Lesson 25) records the failed call with the identity that made it. Grant that role the specific missing permission, scoped to the resources it needs. Granting administrator rights "to stop the errors" turns a deployment fault into a security fault, and the pipeline becomes the most powerful identity in the account.

Bug 3: fast yesterday, slow today

Oversubscription means promising more virtual resources than the physical ones underneath, the overcommitment of Lesson 8. On a private cloud's busy hosts it appears as CPU steal and contention between noisy neighbours. On burstable public cloud instances it appears as performance that falls once burst credits run out. In Kubernetes it shows as nodes whose pods' limits add up to more than the node has, leading to CPU throttling and eviction under pressure. The fix is to measure real demand, set requests honestly, and move steady heavy workloads off shared or burstable capacity.

Bug 4: 429 Too Many Requests

A deployment script calls a provider API in a tight loop and starts receiving HTTP 429. RFC 6585 defines that status code for a user who has sent too many requests in a given time, optionally with a Retry-After header saying how long to wait. Retrying immediately makes it worse. AWS's guidance on timeouts, retries and backoff explains that retries can add load to a system already failing from overload. Clients should therefore back off, increasing the wait between retries, and add jitter, randomness that spreads retries out so clients do not all return at once. Batch calls where the API allows it, and cache answers that do not change.

Bug 5: the disaster recovery test that hit a limit

A recovery drill launches 400 vCPUs of instances in the second region and fails with a limit-exceeded error, although the first region runs that many every day. AWS documents that default quotas are set for each service and, unless otherwise noted, each quota is Region-specific; EC2's On-Demand quotas, for example, cap the vCPUs an account can run in each Region. Some quotas can be raised on request and some cannot, so check and raise them before the day they are needed, as part of the disaster recovery plan from Lesson 4.

Bug 6: works in one region only

A deployment succeeds in one region and fails in another because a managed service or instance type does not exist there. Regional service availability varies; each provider publishes which services run in which regions, as AWS does on its Services by Region page. Check the list when choosing regions, especially the disaster recovery region, which must offer everything the primary uses (Lessons 4 and 26).

Bug 7: the template that used to work

After a cluster upgrade, an old manifest fails with "no matches for kind Ingress in version extensions/v1beta1". The Kubernetes deprecation guide records that the extensions/v1beta1 and networking.k8s.io/v1beta1 versions of Ingress are no longer served as of v1.22, and tells users to migrate to networking.k8s.io/v1. Outdated component definitions take many forms: removed API versions, image IDs that no longer exist, provider plugins pinned years ago, runtime versions past end of support. Keep definitions current with the life-cycle habits of Lesson 24 and test upgrades in staging first.

Bug 8: the service went away

Sometimes the deployment did not change; the platform did. Microsoft announced the deprecation of Azure Application Gateway V1 on 28 April 2023, and V1 retired on 28 April 2026. Microsoft no longer supports V1 resources and they have no SLA. Remaining deployments can experience traffic disruptions as Microsoft blocks the data path and deletes the resources. Google Cloud Deployment Manager, met in Lesson 17, reached end of support on 31 March 2026, and Google directs users to Infrastructure Manager or another deployment technology. Deprecation is announced long in advance through retirement pages and the providers' health and notification services: AWS Health, Azure Service Health and Google Cloud's Personalized Service Health. Someone must read them and own the migration.

Bug 9: incompatibility

A container built for one CPU architecture fails on another with "exec format error". A kubectl far newer or older than the cluster misbehaves; Kubernetes supports kubectl within one minor version, older or newer, of the API server. Libraries, runtimes, drivers and agents all have compatibility matrices, and CrowdStrike's crash was itself a mismatch between content and the code that interpreted it. Check architectures and version-skew rules before a rollout, and let a canary meet real systems before everything else does.

Outages, full and partial

A full outage, where nothing works, is obvious; a partial outage, where one zone, one region or a fraction of requests fail, is harder to see and more common. When failures begin with no change on your side, check the provider's health dashboard early, then use the scope question. If one zone is failing, shift traffic to healthy zones as a multi-zone design allows (Lesson 4); if a dependency is failing, degrade gracefully rather than failing completely.

Bottom line: ask what changed and how wide the failure is, read the exact error, and match it to its category. Then fix the cause, whether a request, a permission, a quota, a definition or a dependency, rather than the symptom. Roll out every fix the way you would roll out any change: tested, staged and reversible.

Common misconceptions

• "Retry immediately until it works." Immediate retries add load to an overloaded service; back off with jitter and honour Retry-After.
• "Grant admin to fix the permission error." Grant the specific missing permission; broad rights create a bigger problem.
• "If it deploys in one region, it deploys in all of them." Quotas, services and instance types differ by region.
• "Nothing changed, so it cannot be our problem." Providers retire versions and services on published dates; unread notices become outages.

The takeaway

• Start with what changed and how wide the failure is, then follow the seven-step method.
• Pending pods, OOMKilled containers and capacity errors point to allocation and sizing; AccessDenied points to permissions; slow shared hosts point to oversubscription.
• Answer 429s with exponential backoff and jitter, and raise region-specific quotas before you need them.
• Check regional availability, keep definitions current, and track retirements such as Application Gateway V1 and Deployment Manager.
• Incompatibilities and outages are found by checking versions, architectures and the provider's health dashboard, and limited by canaries and multi-zone designs.

Sources

1. CrowdStrike. (2024, July 24). Falcon content update remediation and guidance hub: Preliminary post incident review. crowdstrike.com
2. Microsoft. (2024, July 20). Helping our customers through the CrowdStrike outage. Official Microsoft Blog. blogs.microsoft.com
3. Nottingham, M., and Fielding, R. (2012). Additional HTTP status codes (RFC 6585). Internet Engineering Task Force. rfc-editor.org
4. Brooker, M. (2019). Timeouts, retries, and backoff with jitter. Amazon Builders' Library. builder.aws.com
5. Amazon Web Services. (n.d.). AWS service quotas, Amazon EC2 instance type quotas, AWS Services by Region and What is AWS Health? docs.aws.amazon.com (quotas); docs.aws.amazon.com (EC2 quotas); aws.amazon.com (services by region); docs.aws.amazon.com (Health)
6. The Kubernetes Authors. (n.d.). Deprecated API migration guide and Version skew policy. kubernetes.io (deprecations); kubernetes.io (version skew)
7. Microsoft. (n.d.). Application Gateway V1 retirement and migration to V2 and What is Azure Service Health? Microsoft Learn. learn.microsoft.com (V1 retirement); learn.microsoft.com (Service Health)
8. Google Cloud. (n.d.). Cloud Deployment Manager documentation and Personalized Service Health overview. docs.cloud.google.com (Deployment Manager); docs.cloud.google.com (Service Health)

The problem: nothing can connect

At 11:48 PM Pacific time on 19 October 2025, customers of Amazon DynamoDB in the N. Virginia region began seeing errors. They, and other AWS services that depend on DynamoDB, could not establish new connections to it. AWS's post-event summary traced the cause to a latent race condition in DynamoDB's automated DNS management system, which left an incorrect, empty DNS record for the service's regional endpoint. The database was running; its name no longer led anywhere. The disruption to DynamoDB itself lasted until 2:40 AM, but knock-on effects followed: Network Load Balancer health check failures caused connection errors, and new EC2 instance launches failed for hours. The event ended at 2:20 PM on 20 October.

The lesson for anyone troubleshooting a cloud network is that "it is down" is a symptom, not a diagnosis. A connection needs a name that resolves, an address, a route, permission through every filter, a service that answers and, often, clocks that agree. This lesson takes each of those in turn and covers CV0-004 objective 6.2: network service unavailability (DHCP, DNS, NTP, NAT and HTTP status codes), latency, bandwidth and throughput, device misconfiguration, protocol incompatibility and deprecations, IP addressing issues, routing issues and switching issues. It builds on the networking of Lessons 5 to 7 and on Network+ Certification Prep (cs-netplus, CS 215).

Work the path in order

When network services fail

DNS. Name resolution fails in recognizable ways. NXDOMAIN means the name does not exist, SERVFAIL means the resolver could not get an answer, and a timeout means no resolver answered. An empty answer, as in the DynamoDB event, means the record exists but leads nowhere. Caching complicates every fix: resolvers keep answers for the record's time to live, so a corrected record may take that long to reach clients. In a VPC, also check that DNS resolution is enabled and that private zones are associated with the right networks.

DHCP. In cloud networks the provider assigns addresses, but you still control what DHCP tells instances. AWS describes DHCP option sets as controlling the DNS servers, domain names and NTP servers that devices in a VPC use, so replacing an option set with a wrong one can break name resolution or time for every instance that renews its lease. In private and hybrid networks the classic DHCP faults remain: an exhausted scope, a missing relay between VLANs, or a client that has given itself a 169.254.x.x link-local address because no server answered.

NTP. Clocks that disagree break things that look unrelated to time. AWS states that, in most cases, a signed API request must reach AWS within five minutes of the time stamp in the request or it is denied. Microsoft's default Kerberos policy tolerates 5 minutes of clock difference. TLS certificates are checked against the current time, and logs from different systems cannot be correlated when their clocks drift. Each provider offers a time source: EC2 instances can use the local Amazon Time Sync Service at 169.254.169.123, and Compute Engine instances come preconfigured to use NTP. A VPC's DHCP option set can also name NTP servers.

NAT. Instances in private subnets reach the internet through NAT. When they cannot, check that the private subnet's route table sends internet-bound traffic to a NAT gateway, and that the NAT gateway itself sits in a public subnet with a route to an internet gateway. Busy workloads can also run out of NAT ports: AWS notes that each IPv4 address on a NAT gateway supports up to 55,000 concurrent connections to a destination, and Microsoft says most outbound connectivity problems its customers meet come from SNAT port exhaustion and connection timeouts. The fixes are more addresses, connection reuse and closing idle connections.

HTTP. Status codes tell you which side failed.

AWS's load balancer troubleshooting guide gives concrete causes. It lists a 503 when a target group has no registered targets. A 504 appears when the load balancer could not connect to a target within 10 seconds or the target did not respond before the idle timeout. A network ACL blocking traffic can also cause it. A load balancer whose health check probes the wrong port or path marks every target unhealthy and turns a working application into a wall of 503s, a classic network device misconfiguration.

Latency and throughput

Measure before tuning. curl can split a request into its stages, name lookup, TCP connect, TLS handshake and first byte, which shows whether time goes to DNS, distance, encryption or the application. Distance is physics: a user far from the region pays a round trip on every exchange, which is why region choice and CDNs matter (Lessons 4 and 7).

Packet size causes the strangest throughput faults. AWS documents that all EC2 instance types support a 1500-byte MTU and that jumbo frames of 9001 bytes can be used inside a VPC. Traffic over an internet gateway, over VPN connections and between regions without a transit gateway is limited to 1500 bytes. Path MTU discovery normally finds the smallest MTU on a path, but it relies on ICMP messages; when a firewall blocks all ICMP, small requests work while large transfers hang. Throughput has ceilings too: instance size sets network bandwidth, and links such as VPN tunnels have their own limits, so check the provider's documented figures before blaming the application.

Addressing, routing and switching

Scope exhaustion. A subnet holds fewer usable addresses than its size suggests: AWS reserves the first four addresses and the last address of every subnet CIDR block, so a /28 offers 11 of its 16 addresses. Container platforms that give every pod a VPC address can empty a subnet quickly. Size subnets for growth and watch free addresses.

Network overlap. Two networks with overlapping ranges cannot be routed to each other unambiguously; AWS VPC peering is not supported between VPCs with overlapping CIDR blocks. Plan addresses across clouds and on-premises networks before connecting them (Lessons 5 and 6), because re-addressing later is painful.

Routing. A missing route, such as no default route to an internet or NAT gateway or no route back from a peered network, silently drops traffic. A misconfigured route sends it somewhere wrong; remember that the most specific route, the longest prefix, wins, so a 10.20.0.0/16 route overrides a 10.0.0.0/8 route for addresses inside it. At internet scale routing faults are dramatic. On 4 October 2021 a command meant to assess Facebook's backbone capacity took down every backbone connection. Facebook's DNS servers, designed to withdraw their BGP advertisements when they cannot reach the data centers, did exactly that, and the company's names vanished from the internet for hours.

Switching. Private clouds and hybrid edges still run on switches. A wrong 802.1Q VLAN tag puts traffic in the wrong VLAN or drops it. A hypervisor host whose uplink is configured as an access port carries only one untagged VLAN; if its virtual machines need several VLANs, the port must be a trunk carrying those tags (Lesson 5).

Protocol incompatibility and deprecations

Both ends of a connection must share a protocol version. RFC 8996, published in 2021, formally deprecated TLS 1.0 and 1.1, so old clients fail to connect to endpoints that accept only TLS 1.2 or 1.3. OpenSSH 8.8 disabled RSA signatures using the SHA-1 hash algorithm by default, so some old SSH clients and devices could no longer connect to updated servers. Other mismatches are just as common: an IPv6-only client trying to reach an IPv4-only service, or a gRPC service, which needs HTTP/2, placed behind a load balancer listener that speaks only HTTP/1.1. The fix is to upgrade the old end, not to re-enable the deprecated protocol.

Key idea: follow the path, name, address, route, filters, service, protocol and time, and test each link with the right tool. Most network faults are one broken link in that chain, and the symptom usually says which.

Common misconceptions

• "The database is down." Often the name or the route to it is; check DNS and routes before restarting healthy servers.
• "A 503 means the application crashed." At a load balancer it often means no healthy targets, perhaps because the health check is wrong.
• "Blocking all ICMP is safer." It breaks path MTU discovery and leaves large transfers hanging.
• "Time does not matter to networking." Clock skew breaks signed API calls, Kerberos and certificate checks.

Recap

• Work the path: name, address, route, filters, service, protocol and time.
• DNS answers can be missing, empty, failing or stale; DHCP option sets decide DNS and NTP servers; NAT needs a route and enough ports.
• Read HTTP status codes as directions: 4xx toward the client, 5xx toward the server or load balancer, with 502, 503 and 504 meaning different things.
• Check MTU limits and let ICMP through for path MTU discovery; measure latency stage by stage.
• Size subnets for growth, never overlap ranges you will connect, remember longest-prefix routing, and match VLAN tags and port modes.
• Upgrade old protocol versions rather than re-enabling deprecated ones.

Sources

1. Amazon Web Services. (2025). Summary of the Amazon DynamoDB service disruption in the Northern Virginia (US-EAST-1) Region. aws.amazon.com
2. Janardhan, S. (2021, October 5). More details about the October 4 outage. Engineering at Meta. engineering.fb.com
3. Amazon Web Services. (n.d.). DHCP option sets in Amazon VPC, Subnet CIDR blocks, VPC peering basics and Troubleshoot NAT gateways. docs.aws.amazon.com (DHCP); docs.aws.amazon.com (subnets); docs.aws.amazon.com (peering); docs.aws.amazon.com (NAT)
4. Amazon Web Services. (n.d.). Troubleshoot your Application Load Balancers, Network maximum transmission unit (MTU) for your EC2 instance, Set the time reference on your EC2 instance and AWS Signature Version 4 for API requests. docs.aws.amazon.com (ALB); docs.aws.amazon.com (MTU); docs.aws.amazon.com (time); docs.aws.amazon.com (SigV4)
5. Microsoft. (n.d.). Troubleshoot outbound connections in Azure Load Balancer and Maximum tolerance for computer clock synchronization. Microsoft Learn. learn.microsoft.com (SNAT); learn.microsoft.com (Kerberos)
6. Google Cloud. (n.d.). Configure NTP on a VM. docs.cloud.google.com
7. Moriarty, K., and Farrell, S. (2021). Deprecating TLS 1.0 and TLS 1.1 (RFC 8996). Internet Engineering Task Force. rfc-editor.org; OpenBSD Project. (2021). OpenSSH 8.8 release notes. openssh.com

One accepted prompt

In September 2022 Uber published a security update about an intrusion that began with one person's login. According to the company, an attacker had probably bought a contractor's corporate password on the dark web after malware infected the contractor's personal device. The attacker then tried to log in again and again, and each attempt sent the contractor a two-factor approval request that blocked access, until the contractor eventually accepted one. From there the attacker reached other employees' accounts, which gave elevated permissions to tools including G-Suite and Slack, posted a message to a company-wide Slack channel and reconfigured an internal DNS service. Uber said it found no changes to its codebase and no access to customer data held by its cloud providers, and that it was strengthening its multi-factor authentication policies.

Three of the security problems this final lesson troubleshoots appear in that one story: leaked credentials, an authentication method that a tired person could approve, and a path from one account to elevated permissions. This lesson works through security troubleshooting step by step, covering CV0-004 objective 6.3: cipher suite deprecations, authorization issues (privilege escalation and unauthorized access), authentication issues (leaked credentials), software vulnerability issues and unauthorized software. It ends with capstone scenarios that cross all six domains of the exam.

Step 1: contain, but keep the evidence

When a security problem might be an incident, the first moves decide whether you can later explain it. NIST's incident response guidance, SP 800-61 Revision 3 of April 2025, frames incident response within the Cybersecurity Framework's functions, and the cloud versions of its first steps are concrete. Protect the audit trail first, ideally already copied to a separate, locked-down account (Lesson 25). Isolate affected resources rather than deleting them. Move an instance into a security group that allows nothing, and snapshot its disks and keep its logs before anything is terminated. A deleted instance takes its evidence with it. Then revoke what the attacker may hold, sessions, tokens and keys, and only then rebuild.

Step 2: authentication issues and leaked credentials

CISA calls the Uber pattern MFA fatigue, or push bombing: an attacker who has a password triggers hundreds of push prompts in a short time until the user approves one. Its guidance is to implement phishing-resistant MFA (Lesson 25). Where that is not yet possible, it recommends number matching as an interim mitigation, which makes the user type a number shown on the login screen into the app. It also advises training users to report unexpected prompts. For keys, the lasting fix is the one from Lessons 25 and 31: replace long-lived keys with roles and short-lived credentials, so a leak expires on its own.

Step 3: authorization issues, unauthorized access and privilege escalation

Unauthorized access means an identity reached something it should not have: an anonymous request listing a bucket, a developer reading production secrets, a partner account seeing another customer's data. Start from the audit trail to see which identity made the request, then evaluate why the policies allowed it, using tools such as kubectl auth can-i in Kubernetes and the providers' policy analyzers. Common causes are public resources, wildcard permissions and additive role assignments that add up to more than anyone intended (Lessons 25 and 28).

Privilege escalation means an identity uses the permissions it has to gain ones it should not. A classic cloud example is the right to pass roles. AWS explains that to give a role and its permissions to a service, a user must be allowed to pass that role. That lets administrators ensure that only approved users can configure a service with a role. A user who may launch instances and pass any role can launch one with an administrator role and act through it. Kubernetes builds a defense into RBAC: its API prevents users from escalating privileges by editing roles or role bindings. You can only create a role binding if you already hold every permission in the referenced role, or have been granted the bind verb on it. Prevent escalation with narrow PassRole permissions, permission guardrails such as service control policies (Lesson 29), separation of duties, and alerts on every change to IAM.

Step 4: cipher suite deprecations

After a security upgrade, an old client suddenly cannot connect and reports a handshake failure. Protocols and ciphers are retired on purpose: RFC 8996 deprecated TLS 1.0 and 1.1, RFC 7465 prohibits RC4 cipher suites, and OpenSSH 8.8 stopped accepting RSA signatures using SHA-1 by default (Lesson 34). In the cloud the switch is often one setting. AWS describes a load balancer security policy as a combination of protocols and ciphers used to negotiate connections with clients. Choosing a modern policy can therefore lock out an old device in a single change. Diagnose with openssl s_client, offering one protocol version or cipher at a time, and fix the old client by upgrading or replacing it. Re-enabling a deprecated cipher for everyone to rescue one terminal trades a visible fault for an invisible weakness.

Step 5: software vulnerabilities and unauthorized software

When a scanner reports a vulnerability in production, run the cycle from Lesson 27. Confirm the finding, judge its exposure, exploitation evidence and impact, then patch, rebuild or apply a compensating control and verify with a rescan. Unauthorized software is software that arrived outside approved channels: a cryptominer (Lesson 30), a tool an engineer installed for convenience, a container image pulled from an unknown public repository. Find it by comparing what runs with what was approved, using package inventories, image lists and runtime detection, and prevent it with allowlists. NIST's guide to application whitelisting, SP 800-167, describes this approach of permitting only approved applications to run; for containers, admission policies that accept images only from approved registries do the same job.

The point: contain without destroying evidence, then follow the identity: who authenticated, how, and what were they allowed to do? Fix the cause, a leaked secret, a weak MFA method, an over-broad permission, an old cipher or unapproved software, and never restore service by quietly weakening a control.

Capstone: one scenario per domain

The exam mixes the domains, so practise moving between them. For each scenario, decide what you would do before reading the last column.

Before the exam

Lesson 1 described the exam: at most 90 questions, multiple-choice and performance-based, scored from 100 to 900 with 750 needed to pass. Read each objective's opening verb, because "Given a scenario" objectives are tested by asking you to choose, configure or troubleshoot, not to define. In performance-based questions, read the whole task before changing anything and remember that you can skip and return. Check CompTIA's Cloud+ page before booking, because a new version, CV0-005, is in draft. No course can promise a pass; what this one can promise is that nothing on the CV0-004 objectives list should be new to you.

Common misconceptions

• "Any MFA stops account takeover." Push approvals can be bombed until someone accepts; phishing-resistant MFA, or number matching in the meantime, closes that gap.
• "Deleting the leaked file fixes the leak." The secret lives on in history and copies; rotate it.
• "Terminate the compromised instance immediately." Isolate and capture evidence first, then rebuild.
• "Re-enable the old cipher so the device works." Upgrade the device; a deprecated cipher weakens every connection that can use it.

What to carry forward

• Contain without destroying evidence: protect logs, isolate, snapshot, revoke, then rebuild.
• Leaked credentials call for revocation, rotation and an audit-trail review; push bombing calls for phishing-resistant MFA or number matching.
• Unauthorized access and privilege escalation are found from the identity in the audit trail and prevented with narrow permissions, guardrails and Kubernetes' built-in escalation checks.
• Fix cipher suite deprecations by upgrading old clients, run the vulnerability cycle for software issues, and allowlist approved software and images.
• Across all six domains, the habits are the same: know what normal is, change things through code and pipelines, give every identity only what it needs, and verify every fix.

Sources

1. Uber. (2022, September 16). Security update. Uber Newsroom. uber.com
2. Cybersecurity and Infrastructure Security Agency. (2022). Implementing number matching in MFA applications [Fact sheet]. cisa.gov
3. National Institute of Standards and Technology. (2025). Incident response recommendations and considerations for cybersecurity risk management: A CSF 2.0 community profile (NIST Special Publication 800-61 Revision 3). csrc.nist.gov
4. Sedgewick, A., Souppaya, M., and Scarfone, K. (2015). Guide to application whitelisting (NIST Special Publication 800-167). National Institute of Standards and Technology. csrc.nist.gov
5. Amazon Web Services. (n.d.). Grant a user permissions to pass a role to an AWS service and Security policies for your Application Load Balancer. docs.aws.amazon.com (PassRole); docs.aws.amazon.com (TLS policies)
6. The Kubernetes Authors. (n.d.). Using RBAC authorization: privilege escalation prevention. kubernetes.io
7. Popov, A. (2015). Prohibiting RC4 cipher suites (RFC 7465). Internet Engineering Task Force. rfc-editor.org
8. CompTIA. (n.d.). CompTIA Cloud+ and CompTIA Cloud+ CV0-004 certification exam: Exam objectives version 2.0. comptia.org; comptiacdn.azureedge.net