🚨 OpenBao Incident Response & Rotation Playbook
When a compromise occurs, your response depends on the "blast radius" of the leaked credential. This playbook covers the four layers of compromise in the OpenBao environment and the exact steps to mitigate and rotate them.
Layer 1: A Specific Secret is Compromised
Scenario: A developer accidentally commits a database password (
my-app-secret) to a public GitHub repository, or a specific database is breached.
Response Strategy:
- Rotate at the Source: OpenBao only stores the secret; it doesn't control the target system. Immediately change the password in the actual target database/API.
- Update OpenBao: Log into the OpenBao Web UI and update the value in
lab-shared/my-app-secretwith the new password.
*Note: Because we use the KV-v2 engine, OpenBao automatically versions the secret. The old, compromised password becomes "Version 1", and the new password becomes "Version 2". - Redeploy Applications: Restart any microservices relying on that secret so they authenticate with OpenBao and pull down "Version 2".
- Audit: Check the
audit.logon the host machine to see what other IPs or users read that specific secret recently.
Layer 2: Machine Identity Compromise (AppRole)
Scenario: A Python script's
RoleID and SecretID are found hardcoded in a script or leaked by a disgruntled contractor.
Response Strategy:
-
Revoke Active Sessions: Immediately kill any active login sessions tied to the shared AppRole. This stops the bleeding instantly.
bao lease revoke -prefix auth/approle/role/lab-shared-app/ -
Destroy the Leaked SecretID: If you know the exact SecretID that was leaked, destroy it. (If you don't, you can destroy all custom SecretIDs and force developers to generate new ones).
bao write auth/approle/role/lab-shared-app/secret-id/destroy secret_id="<leaked_secret_id>" - Rotate: Have the legitimate application owner log into the Web UI Browser CLI and run
write -f auth/approle/role/lab-shared-app/secret-idto generate a brand new, clean SecretID, and update their server environment variables. - Audit: Search the
audit.logfor the compromisedrole_id. Identify every secret path that machine accessed while compromised, and treat those secrets as a Layer 1 Compromise (rotate them all).
Layer 3: Human Identity Compromise (OIDC/SSO)
Scenario: A user's Google Workspace account is hijacked, or their laptop is stolen while they are logged into OpenBao.
Response Strategy:
- Cut off the Source (Google Workspace): Immediately suspend the user's account in your Google Workspace admin console. Because OpenBao relies on OIDC, the attacker cannot initiate any new logins.
-
Revoke OpenBao Tokens: The attacker might have an active session token sitting in their browser cache. Revoke all active OIDC tokens in OpenBao to force an immediate kick:
bao lease revoke -prefix auth/oidc/ - Audit the User's Actions: Search the
audit.logfor the user's email address. Look for anykv getor read operations during the time of the compromise. Treat every secret they viewed as compromised and initiate a Layer 1 rotation for each of them.
⚠️ Layer 4: System-Level Compromise (Master Keys)
Scenario: An attacker gains root access to the
lab-microservices host server, discovers a saved Initial Root Token, or finds 3 of your 5 Unseal Keys stored in a compromised password manager.
If the Root Token or Master Unseal Keys are compromised, the entire fortress is considered breached.
Response Strategy:
-
Revoke the Root Token: If a root token was leaked, revoke it immediately. (You should rarely have active root tokens anyway!).
bao token revoke <leaked_root_token> -
Rekey the Vault (Crucial): If your Unseal Keys are compromised, you cannot simply "change a password." You must generate an entirely new master encryption key and cryptographically shred the old one. This is called a Rekey.
# Initiate the rekey process bao operator rekey -init -key-shares=5 -key-threshold=3OpenBao will prompt you to enter 3 of your old, compromised unseal keys to authorize the action. Once provided, it will generate 5 brand new unseal keys.
- Distribute and Destroy: Securely distribute the new unseal keys to your administrators, and permanently delete all records of the old keys.
- Total Audit: Because the attacker had god-mode access, you must assume everything is compromised. Review the
audit.logextensively, and systematically rotate every critical secret stored in the lab.