Threat intel
Northflank breach: assume every variable was decrypted
Northflank told customers to treat all environment variables as stolen after a Wiz alert, and AI keys were abused. What to rotate first and what to check.
Northflank, a platform as a service (PaaS) that builds and runs containers, databases and jobs for its customers, received a threat alert from the cloud security firm Wiz at 11:43 UTC on October 30 and posted a security incident at 13:40. At 13:58 its status page told every customer to "assume all environment variables have been decrypted" and to disable or rotate all keys. By 23:38 UTC users were reporting that AI API keys stored on the platform were being used by someone else.
If you have a Northflank account, this applies to you. Northflank has not said how the attacker got in, how many customers were affected, or which projects or regions. Wiz counts Northflank as the ninth AI developer platform to disclose an incident in October, after Railway, Zeabur and OpenCode among others, and it has watched keys stolen in these breaches reach resellers within a week.
What Northflank held
Northflank keeps secrets as runtime variables and build arguments on each service and job, in secret groups shared across a project, as global secrets, as secret files mounted into containers, and as template argument overrides. Values are encrypted at rest and decrypted just before injection into a build or container. Telling customers to treat them as decrypted means Northflank believes the attacker reached that step, or the keys behind it.
Northflank's first fix was a network policy patch at 12:02 UTC, 19 minutes after the alert, followed by more at 12:50 and 13:05, and it called in Wiz's incident response team at 13:00. A Kubernetes network policy decides which pods may talk to which, so this suggests something could reach a service it should not have, but Northflank has not confirmed the entry point.
Addons, the managed Postgres, MongoDB, MySQL, Redis, MinIO and RabbitMQ instances Northflank runs, fall under the same advice, and Northflank asked customers to switch off public access on any addon that has it.
Bring your own cloud (BYOC) customers run Northflank's clusters in their own cloud account. Northflank's BYOC page says workloads, databases and secrets stay in the customer's virtual private cloud (VPC) and its control plane handles only orchestration metadata. Yet the status page makes no exception for BYOC, and the incident banner on that same page says to rotate all user defined credentials. Rotate BYOC secrets too, and check your cloud audit trail for actions taken with the access you granted Northflank.
What happens to stolen keys
Northflank has published no figures on abuse. It relays user reports of unauthorised AI key use, says they may fit a broader pattern of attacks on AI credentials, and points to Wiz's "Black October" research. BleepingComputer's questions to Northflank went unanswered.
Wiz describes two stolen Amazon Bedrock keys from these October breaches without naming the platform behind either, so treat them as the general pattern. Each got one test call within 30 minutes of the vendor breach, then 14 to 120 calls a day from the same small Go client. One went to bulk use about four days later and made about 800,000 calls in roughly 30 hours. The other turned up in what looked like a reseller pool after about seven days and made about 1.7 million calls in roughly 13 hours. Wiz puts the two together at more than $100,000 of usage at list price.
The second key did less damage because its IAM policy allowed only us.* inference profiles, so most of the flood hit regional quotas and was throttled. The first allowed global.* cross region profiles, which spread the load and let all but about 6% of calls through. Wiz also lists four IP addresses seen checking or using stolen keys in October: 23.106.46.239, 38.246.114.118, 104.243.32.50 and 31.57.201.201. Wiz has not tied them directly to the platform breaches.
A key revoked during those days of validation calls has cost very little.
Rotation order for Northflank customers
Northflank has not said when the attacker got in, so start log searches well before the alert; Wiz saw its indicator addresses from October 18, and we use October 1. For every external key, create the new one, deploy it, confirm traffic, then revoke the old one at the issuer. Deleting a variable in Northflank revokes nothing.
1. AI provider and cloud keys. OpenAI, Anthropic, Google, OpenRouter and any AWS access key with Bedrock rights come first, because they were the target and turn straight into spend. Where a workload can use an IAM role instead, switch now so no long lived AWS key goes back into Northflank.
2. Database credentials. Use the addon rotation Northflank enabled during the incident: Postgres and MongoDB on October 30, MySQL and Redis at 13:34 UTC on October 31, MinIO and RabbitMQ later that night. In the API it is two calls. POST /v1/projects/{projectId}/addons/{addonId}/secret-rotation issues new credentials while the old ones keep working; redeploy the services that use them, then call .../secret-rotation/finalise to remove the old ones. Redis is different: it restarts and revokes the old password at once, so redeploy linked services straight away. MinIO changes only the secret key, which breaks presigned URLs signed with the old one. For databases you run elsewhere, change the password inside the database, not only the DATABASE_URL.
3. Pipeline tokens. GitHub and GitLab tokens, container registry credentials and the Northflank API tokens your CI uses. Northflank's API has a call to rotate the key on an API token.
4. The rest. Payment keys, webhook secrets, OAuth client secrets and session signing keys. A new JWT signing key logs every user out, so pick the time.
Checking for use
On each AI provider's usage page, filter by key from October 1 and look for models your code never calls, traffic at hours your team is offline, or a jump after a few quiet days. Report anything suspicious to Northflank support and to the provider.
For AWS keys, aws iam get-access-key-last-used --access-key-id <id> shows the last service and region. In CloudTrail, filter on the old access key ID and look for listing of Bedrock models or service quotas, which came just before the bulk traffic in both Wiz cases, and for the four addresses above.
In Northflank, read the audit log for API tokens, team members and secret groups you did not create. Northflank turned audit logs on for all customers, free plans included, at 01:04 UTC on October 31. Accounts without them before may have no record of earlier changes.
Cap what the next leak can cost
Encryption at rest did not help, because a deployment platform has to decrypt secrets to inject them. Limit what each key can do instead. Give every service its own AI key in its own provider project or workspace with a monthly spend limit, so a stolen key has a ceiling and the usage page names the service it came from. For Bedrock, scope IAM policies to the models and inference profiles you call, and leave out global.* unless you need it.
Move what you can out of the platform's variables into a secret manager such as AWS Secrets Manager or HashiCorp Vault, fetched at start up with a short lived identity where your cluster supports one, as a BYOC cluster in your own account can. A dump of the platform's variables then yields only references to secrets. Northflank Enterprise customers can also supply their own key management service (KMS) key for stored secrets, which keeps the master key in an account they control and can revoke.
Our cloud and Kubernetes security team reviews where deployment platforms and pipelines hold secrets, and our platform and cloud engineering team moves them into a secret manager with short lived credentials. Open the chat and Yaali, our AI agent, will pass your question to an engineer.
Sources: Northflank status page, security incident, Northflank status, BleepingComputer, Wiz, Black October for AI developer platforms, Qovery, Northflank BYOC, Northflank security documentation, Northflank API, rotate addon secrets, Northflank API, finalise addon secret rotation, Northflank API, rotate a key on an API token, AI TLDR, FourWeekMBA.