P.K. SHARMA

Cyber security intelligence, AI governance, practitioner analysis

Cloudflare skipped block zeroing for speed, and researchers read 5,614 directory blocks that were not theirs

A dm-thin pool option meant deleted container disks went back into a shared pool without being cleared. Across six production placements, not one testable directory block belonged to the researchers who triggered the allocation.

By Parminder Kumar Sharma · · 12 min read

Editorial illustration for the briefing: Cloudflare skipped block zeroing for speed, and researchers read 5,614 directory blocks that were not theirs

Five thousand six hundred and fourteen blocks, and not one of them theirs

Across six production placements on Cloudflare's container platform, security researchers at Accomplish examined 5,614 ext4 directory blocks that they could test for ownership. Not one of those blocks belonged to the filesystem they had created. Checksum analysis identified 2,700 distinct directory inodes belonging to somebody else.

Those blocks arrived on a disk Cloudflare had just handed them, on infrastructure sold on the promise that one customer's workload cannot see another's. Cloudflare published the postmortem on 24 September 2026 and says the vulnerability is fully remediated, that no customer action is required, and that it found no evidence of malicious exploitation.

Here is what that number does not establish. It does not establish that anybody's data was taken, because the only activity Cloudflare could attribute to the technique was the researchers' and its own engineers'. It does not establish that an attacker could have chosen a victim, because the platform picks the host and the pool decides which freed blocks come back. It does not establish how many customers had blocks in that pool, because Cloudflare does not say and the researchers could not have known. And it does not establish that the researchers read anybody's content: they wrote tooling that counted and classified blocks rather than printing what was in them.

What it does establish is narrower and worse. On the machines they were placed on, the residue was not an edge case. It was the normal condition of a freshly allocated block.

One optional feature argument, set on the pool

Cloudflare Containers give each container a writable root disk carved out of a Linux device mapper thin pool. Each container runs inside its own virtual machine under the Firecracker monitor, and Firecracker presents that disk to the guest as /dev/vdc. Thin provisioning allocates physical storage only when a virtual disk writes to a region that has not been mapped yet, which is what makes it cheap. The affected pools used a 64 KiB thin block.

When a container's thin volume was deleted, its physical blocks went back to a pool that, in Cloudflare's words, "served workloads belonging to multiple customer accounts". That is ordinary. What was not ordinary was the pool configuration, which carried one option:

skip_block_zeroing

Cloudflare describes what it does: "With this option configured, dm-thin skips zeroing newly allocated blocks before making them accessible." The kernel's own thin provisioning documentation describes the same option in a single sentence, "Skip the zeroing of newly-provisioned blocks", in a list of optional feature arguments, with no stated security implication and no stated default. The default is inferred from the existence of an option to turn it off.

Reading an unmapped region does not expose anything: dm-thin returns zeroes without allocating a physical block. The exposure needs an allocation. So the proof of concept found 64 KiB aligned regions matching free space in the guest's own ext4 filesystem and wrote a single aligned 4 KiB block into each one. That write forced dm-thin to allocate a recycled 64 KiB physical block, replaced the first 4 KiB of it, and left the other 60 KiB holding whatever the previous owner had left. A raw read of /dev/vdc then returned bytes the new container had never written.

Sixty KiB of sixty four is 93.8 per cent of every block touched this way.

A diagram of one 64 KiB thin provisioned block passing between two tenants. A deleted container's blocks return to a pool shared across accounts, and because the pool skips block zeroing they keep their contents. The next tenant writes one aligned 4 KiB block, which forces allocation and overwrites only the first 4 KiB, leaving 60 KiB readable. A second panel shows that zeroing only affects new allocations, so running disks and cached image layers had to be retired as well.
Drawn from Cloudflare's postmortem of 24 September 2026.

How they proved the blocks were somebody else's

Recovering bytes is easy to claim and hard to attribute. A block full of plausible looking filesystem structure might be your own, left over from your own test. The researchers solved that with a property of ext4 itself.

When ext4 runs with the metadata_csum feature, the checksum on a directory block incorporates values tied to the filesystem and the inode. A directory block therefore carries, in effect, a fingerprint of the filesystem it came from. Blocks that check out against your filesystem are yours. Blocks that do not are not.

They validated the method before trusting it. Against blocks they had deliberately created and then deleted inside their own controlled test filesystem, the method correctly attributed all 162 of them. Only then did they apply it to production placements.

What the checksum analysis returned, as reported in Cloudflare's postmortem

MeasureResult
Testable directory blocks across six production placements5,614
Of those, attributed to the researchers' own filesystem0
Distinct foreign directory inodes identified2,700
Control test on blocks they created and deleted themselves162 of 162 correctly attributed
Placements where residual material was observed18 of 24
Underlying nodes where residual material was observed20 of 22
Continents covered by those nodes4

The recovered block types, again in Cloudflare's words, "included directory structures, database pages, and structurally complete SQLite databases". A structurally complete database is not a fragment. It is a file that opens.

The handling deserves noting, because it is the part that makes the disclosure usable. The researchers reported using scripts that output only aggregate counts and format checks rather than recovered file contents. The materials submitted to Cloudflare contained no third party filenames, identifiers, credentials, hostnames, addresses or recovered content values, and the researchers confirmed afterwards that the recovered data was kept confidential and securely deleted. Cloudflare says the post was prepared in collaboration with them.

What the exposure was not

Cloudflare is precise about the limits of the technique, and those limits are real. They are also narrower than the reassurance they will be read as.

Stated in the postmortem against not established by it

StatedNot established
An attacker could not select a particular victim, workload, host or dataThat the blocks recovered were harmless, since they included complete SQLite databases
Exposure depended on Cloudflare's placement and which freed blocks dm-thin reassignedAny figure for how many accounts had blocks in the affected pools
The researchers did not demonstrate modification of another customer's active dataThat modification was impossible, only that it was not shown
No impact to workload availability was demonstratedAny statement about data written before the telemetry Cloudflare retains
Exploitation required a Workers Paid accountThat the account requirement is a meaningful barrier, given the price of one
No evidence of malicious exploitation in retained disk I/O telemetryThat none occurred outside the retention window

Six hours for the flag, fifteen days for the blocks

The response was fast in the way that gets praised and slow in the way that matters, and the gap between the two is the most useful thing in the postmortem.

Removing skip_block_zeroing restored the kernel default of clearing a block before handing it over, and the researchers independently confirmed their proof of concept stopped working. But zeroing applies to blocks allocated from that point on. It does nothing to blocks already mapped into thin devices that exist. Cloudflare spells out where those were: in running container disks, and in each host's cache of prepared dm-thin snapshots for OCI image layers. A new container could inherit mappings from a cached layer without allocating those blocks again, so residual bytes in unused regions, including ext4 free space, stayed readable through a raw read of the device.

So the second half of the fix was operational rather than clever. Retire every running container disk. Remove cached image snapshots created before the mitigation. Drain hosts during off peak hours, restart the virtual machines on each one, and clear each host's image cache so that disks and cached layers are rebuilt from zeroed allocations.

Cloudflare's own timeline, with elapsed time from the report computed for this briefing

StepTime, UTCElapsed from the report
Oren Yomtov of Accomplish reports through HackerOne4 September, 15:26zero
Security incident opened, production setup confirmed4 September, 18:453 hours 19 minutes
Runtime fix and its reuse test merged4 September, 21:276 hours 1 minute
Changes merged for new and live pools4 September, 22:036 hours 37 minutes
Rollout begins4 September, 23:157 hours 49 minutes
Rollout complete, clearing of old pool data begins7 September, 06:132 days 14 hours
Researchers report the proof of concept has stopped working14 September, 10:509 days 19 hours
Bounty awarded14 September, 12:529 days 21 hours
Cleanup of all pre mitigation cached snapshots complete19 September, 15:0314 days 23 hours
Postmortem published24 September20 days

Read the two halves separately. The configuration change that stopped the technique landed in six hours and one minute. The work that removed the data anybody could still reach took until 19 September, fifteen days after the report. For the twelve days between the rollout completing on 7 September and the cleanup finishing on 19 September, the technique no longer worked while the residue was still there, reachable by any container that inherited a pre mitigation mapping.

That is not a criticism of the schedule. Draining hosts across a global fleet at off peak hours is the responsible way to do it. It is an argument about what "patched" means on shared infrastructure, and about which date belongs in your own incident record if you were running on it.

How Cloudflare looked for anyone else doing it

The technique leaves a shape in the telemetry. A 4 KiB write reaches an unmapped region, triggers allocation of a recycled 64 KiB block, and a subsequent read returns substantially more data than the container ever wrote. That write to read ratio is characteristic, and Cloudflare turned it into detection signatures using the researchers' proof of concept and its own reproduction as reference activity.

Applied to the historical disk I/O telemetry it retains, those signatures found the researchers and Cloudflare's own engineers, and nothing else. The company states: "We saw no evidence that this specific attack vector was exploited by anyone else."

The boundary you bought and the boundary you got

Tenant isolation, as it is sold, is a stack of comforting nouns. Container. Sandbox. Dedicated virtual machine. Firecracker. Every one of those was present and working here. The isolation that failed was none of them: it was whether the block allocator cleared 64 KiB of storage before giving it to the next customer, governed by an optional feature argument in a device mapper pool, set for performance, and visible in no product tier, no dashboard and no contract.

That is the friendly name fallacy in its purest form. A customer who asked "are containers isolated from each other?" would have been told yes, correctly, by everyone at every level of the company. The question that would have found this is "which optional arguments are set on the storage pools those containers are carved from, and who reviews them?"

The second lesson is subtler and generalises further. The fix and the exposure were decoupled. Turning zeroing back on was correct, immediate and insufficient, because caching had already copied the problem forward: a host's cache of prepared image layer snapshots could hand a new container mappings that predated the fix. Anywhere you warm a filesystem and reuse it, a change to allocation behaviour does not reach backwards. Golden images, warm pools, prepared snapshots and layer caches all have this property.

What to do about it

Cloudflare customers have nothing to apply. The rest of this is for anyone who runs multi tenant infrastructure of their own, or who buys it.

Take this with you

In the order worth doing

  • Check every thin provisioned pool you run for skip_block_zeroing, including pools created by orchestration tooling or vendor defaults you did not set yourself.
  • If it is set, establish whether you set it deliberately, what it was measured to save, and whether that measurement still exists.
  • Treat any fix that changes allocation behaviour as incomplete until you have accounted for storage already mapped to live workloads.
  • Enumerate everything that hands a prepared filesystem to a new workload: image layer caches, golden images, warm pools, snapshots, restored volumes.
  • Ask your provider, in writing, what disk level telemetry they retain and for how long, because that retention is the ceiling on any statement they can later make about exploitation.
  • For data you place on shared infrastructure, assume deleted means unlinked rather than erased, and encrypt at rest inside the workload where the data warrants it.
  • If you run a disclosure programme, publish the date the technique stopped working and the date the exposure was removed as two separate dates.

The question this leaves

Cloudflare did the things you would want done. It confirmed the report in three hours, merged a fix in six, published a postmortem with its own timeline, named the researchers, paid the bounty and wrote down what it could not rule out. The disclosure is better than most and the handling by Accomplish was better than most. None of that is in question.

What is in question is how the flag came to be set, and the postmortem does not say. It does not say when skip_block_zeroing was added to those pools, who decided, what it was worth in throughput, or whether anybody reviewed it after Containers moved from preview to a paid product carrying other people's databases. Those are the facts that would tell you whether this was a stale performance choice or a live tradeoff nobody revisited, and they are the facts most likely to be true of somebody else's platform right now.

Every provider selling isolation has a storage pool behind it. What would yours say if you asked which optional arguments are set on it, and when they were last reviewed?

Sources

  1. PrimaryCloudflare's own postmortem, used for every figure, quote, mechanism and timeline entry in this briefingCloudflareaccessed 2026-09-25
  2. PrimaryDevice mapper thin provisioning, used for what skip_block_zeroing changes and the default behaviour it overridesThe Linux Kernel documentationaccessed 2026-09-25
  3. Reported bySecondary coverage, checked against the postmortem and cited only where this briefing notes a claim the postmortem does not makeThe Hacker Newsaccessed 2026-09-25

Share this briefing

Know someone who owns this problem? Send it to them.

Related briefings

The briefing, in your inbox

Practitioner analysis of cyber and AI security news. No vendor noise.

One email per briefing. Unsubscribe any time.