What the COW and COW Filesystem

Introduction

I write this from a painful personal experience, an experience I never even thought about, let alone how to handle it when you can no longer Copy-on-write. COW filesystems like BTRFS are great for data integrity and snapshotting, but they have quirks. In this blog, we'll explain COW, why you need to keep some free space on your disks, and the painful process of deleting files when your filesystem is almost full.

What Is Copy-on-Write (COW)?

COW means that the system doesn't immediately overwrite the original data when you modify a file. Instead, it writes the changes to a new block and updates the metadata to point to the new location. This approach improves data safety and enables features like snapshots, which means every change is an additional write.

Why Reserve Free Space?

With COW filesystems, deletions and updates are not as simple as just marking blocks free; they involve extra writes. For BTRFS, leaving at least 10% of your disk space reserved is widely recommended. This reserved space is crucial because:

• Metadata Updates: COW operations require writing new metadata. If the disk is too full, even deleting a file can fail because there isn't enough room to store the latest copy of metadata.

• Performance Overhead: When space is scarce, the system struggles with fragmentation and slower write speeds, making recovery from an full disk excruciating.

The Pain of a Full COW Filesystem

I learned this the hard way: when a BTRFS filesystem fills up, you can't simply delete files like you would on a traditional filesystem. Since deletion in a COW system is a write operation, you may face a scenario where deleting one file doesn't free enough space to allow the metadata update needed to delete the following file.

A practical workaround is to delete files in a stepped approach:

• Start Small: Remove 1 small file.
• Double Up: Then delete 2 small files.
• Keep Scaling: Follow with 4, then 8 small files at a time until there is room to delete larger files.

This incremental removal gradually frees up the necessary space for the COW mechanism to do its job. It's a slow, frustrating process that can take a hour of file-by-file deletion simply to reclaim enough space to recover from a near-full disk.

Technical Insights into BTRFS and COW

BTRFS leverages COW for features like:

• Snapshots and Subvolumes: This allows you to capture the state of the filesystem at any point in time.
• Checksumming: Ensuring data integrity by keeping track of errors.

However, the same mechanisms mean that BTRFS needs extra free space to effectively manage its metadata and data copies. When the disk is overly filled, every deletion or modification can trigger a cascade of additional write operations that worsen the situation, potentially locking you into a cycle of manual space recovery.

Final Thoughts: Hope the COW

If you're using a COW filesystem like BTRFS, remember:

• Keep It Light: Don't fill your disk beyond 90% - keep that 10% free for smooth operations.
• Plan for Recovery: If you fill up your disk, be prepared for a painstaking process of incremental deletions (1, 2, 4, 8…).
• Store Metadata on a separate disk, yes that's an option!

Ultimately, the experience taught me a valuable lesson—take care of your COW, or it will take care of you in the most unexpected (and exasperating) ways.

And as they say in the world of COW: "Hope the COW doesn't copy on write itself into a corner!"