Resilience

On October 8 Canonical announced on Ubuntu Discourse that its websites were "under a sustained distributed denial of service attack". A distributed denial of service (DDoS) attack floods a service with traffic from many machines until real users cannot reach it. The attack took down ubuntu.com, canonical.com, the Ubuntu blog, the developer site, and the Ubuntu Security Notices pages and Security API. OMG! Ubuntu reports the main sites were offline for 4 hours 12 minutes. On October 9 Canonical said most services were available again, and on October 14 it posted that all websites were restored.

Package installs kept working this time. Security Affairs reports that the package repositories and mirrors stayed up, along with Launchpad, the Snap Store, the image servers and Ubuntu Single Sign-On (SSO). What failed is the advisory side: any scanner, dashboard or script that asks Canonical's Security API which Ubuntu Security Notices (USNs) apply to your packages got no answer. This was the second attack on Canonical this year, and the first one did stop some machines from updating.

What each attack hit

Ubuntu's software is spread over very different infrastructure. The APT archive, the store of .deb packages that apt installs from, is copied to mirrors around the world run by universities, hosting firms and cloud providers, so a flood aimed at Canonical's own servers barely touches it. Launchpad, which hosts Personal Package Archives (PPAs), the Snap Store, Livepatch and the security API run on Canonical's own infrastructure with no outside copies, so an attack on Canonical can take them offline.

The October attack went for the web front ends. Besides the sites above, Security Affairs lists assets.ubuntu.com, academy.canonical.com, jaas.ai and maas.io, and says the Security API and Livepatch, Canonical's service that applies kernel fixes without a reboot, were disrupted for a time. Canonical has not said how large the attack was or where the traffic came from.

The earlier attack reached further. OMG! Ubuntu dates its start to about 6 PM UK time on April 30, and Canonical's status page marked it resolved on May 4, with Launchpad still patchy afterwards. Other outlets describe it as a May attack, so you will see both dates. Canonical called it a "sustained, cross-border attack". Launchpad and its PPAs, the Snap Store, Ubuntu SSO, the mailing lists, security.ubuntu.com, the Livepatch API and Landscape were all affected. The main archive held because of its mirrors, but Security Affairs and WebProNews both report that some users could not install updates.

Who is behind it

The Islamic Cyber Resistance in Iraq, also called 313 Team, claimed the spring attack, according to Security Affairs, WebProNews and Tom's Hardware. WebProNews says the group announced it on Telegram and demanded a virtual meeting with Canonical executives, threatening more attacks. Canonical has not confirmed the claim.

As of October 12 nobody had claimed the October attack. Security Affairs notes that it is unclear whether it continues the same campaign or comes from someone else, and nothing published so far settles that.

What it means for your patching and CI

A few hours without ubuntu.com is a nuisance. For a team that runs Ubuntu, the risk is that patch and build pipelines quietly depend on Canonical services that do not have mirrors. These are the places to look, roughly in order of how often they break builds.

Put a cache between your machines and the archive

If every CI runner and server pulls straight from archive.ubuntu.com or a single cloud mirror, an upstream problem becomes your problem on every host at once. An apt caching proxy such as apt-cacher-ng (port 3142 by default) or a full local mirror serves packages you have already fetched and cuts outbound traffic too. Point clients at it with Acquire::http::Proxy "http://apt-cache.internal:3142"; in a file under /etc/apt/apt.conf.d/.

In the same directory, set Acquire::Retries "5"; so apt-get update and apt-get install retry a failed download instead of giving up on the first timeout. For CI jobs, wrap the apt step in a retry with backoff (for example 30, 60 and 120 seconds) and fail with a clear message naming the upstream, so nobody spends an hour debugging their own proxy.

Stop fetching PPAs and snaps at build time

The spring attack took down Launchpad and the Snap Store while the archive stayed up, and Launchpad was still patchy days later. If a Dockerfile runs add-apt-repository ppa:... or a pipeline runs snap install, that build fails with them. Copy the PPA packages you depend on into an internal repository or a base image, and pin exact versions with an apt preferences file so a rebuild installs what you tested. For snaps, keep the .snap and .assert files that snap download fetches, and install them offline with snap ack followed by snap install.

Have a second way to read Ubuntu Security Notices

During the October outage, tools that query the Security API or scrape ubuntu.com/security/notices had nothing to read. The important part is what your tooling does then. If a failed lookup is reported as "no vulnerabilities found", the dashboard shows green while you are blind. Make the scanner report "data unavailable" and alert on it.

Keep at least one other channel:

• The ubuntu-security-announce mailing list on lists.ubuntu.com sends every USN by email. The lists were down in spring but up in October, so it is a backup and not a guarantee.
• Canonical publishes the USN data as OVAL files, a machine-readable format that OpenSCAP and other scanners understand, at security-metadata.canonical.com/oval/com.ubuntu.<release>.usn.oval.xml.bz2. Keep the latest copy in your own storage, so scans can run against yesterday's data during an outage.
• Each updated package carries its own changelog, which names the CVE IDs it fixes. After an upgrade, zcat /usr/share/doc/<package>/changelog.Debian.gz | head -40 shows them without contacting Canonical.

Know where to look first

Canonical posted updates at status.canonical.com and on Ubuntu Discourse during both attacks. Put both links in the runbook for whoever is on call, together with the list of your jobs that depend on Canonical services, so that a failing build can be matched to a vendor outage in minutes.

The same lesson for your own sites

Canonical's archive survived both attacks because it is mirrored widely, away from the servers behind its websites. The same split works for a smaller organisation: put the marketing site, the customer portal and any download or update service your customers' machines call on separately protected infrastructure, so a flood against one does not take the others down.

For the sites themselves, the basics from our NightmareStresser post still apply: upstream scrubbing or a content delivery network in front of public sites, rate limits on login, search and API endpoints, and a runbook with your provider's emergency contacts and the person who can approve paid mitigation. Canonical's October statement gave no size, origin or method, so there is nothing more specific to tune for yet.

Our platform and cloud engineering team builds CI pipelines and package caches that keep working when an upstream goes down, and our security operations team watches for both traffic floods and silent gaps in vulnerability data. Open the chat and Yaali, our AI agent, will pass your question to the engineer who would do the work.

---

Sources: Ubuntu Discourse, Canonical announcement, Canonical status page, OMG! Ubuntu, October attack, OMG! Ubuntu, spring attack, Security Affairs, Phoronix, WebProNews, Tom's Hardware, Ubuntu OVAL data, ubuntu-security-announce.