Cleaning duplicate data to make backups lighter

Storage is the bit that quietly eats a homelab. Drives wear out, power adds up, and messy layouts make restores slower than they should be. I wanted less wasted space, shorter backup windows, and a setup I could actually trust when something goes wrong.

Introduction

Importance of storage optimisation

Storage was the biggest ongoing cost in my lab. The bill is not just the drives. It is the power draw, the rebuild time, and the hassle of working out where data lives when you need it back. I care about usable capacity, power draw, and recoverability. If those three are under control, the lab stays useful instead of becoming a pile of disks and regret.

Common challenges with duplicated data

Duplicates turn up everywhere. Photo libraries, VM snapshots, and automated media grabs all love making extra copies. In my case I started with about 50 TB raw across five arrays, including parity, two backup arrays, and a lot of duplicated content. The result was simple: space was wasted and backup windows stretched out.

Overview of backup strategies

My plan was three replicated copies of the core 4–5 TB of primary data, plus a separate backup array sized for about 10 TB. That gives quick local restores and a second copy for disaster recovery. Pick a policy that matches how long you can tolerate downtime and how much you want to spend on storage and power.

Strategies for Cleaning Duplicate Data

Identifying duplicate data sources

Start by checking where the big data lives. Make a list of services that hold large files: media servers, photo sync tools, VM stores, and old archive folders. Run simple checks:

• Find the largest directories with du -sh /* and work down from there.
• Use find to spot old snapshot chains.
• Check application configs, for example Immich and Jellyfin, to see where originals and transcodes are stored.

Look for patterns. If a folder contains lots of identical files across date stamps, it is probably duplication from a sync tool. If VMs have frequent snapshots, those snapshots may be holding full disk copies.

I found two arrays holding the same media collection because one had started as the primary and the other was a casual copy from years earlier. Deleting that copy freed several terabytes.

Tools for data management

Use tools that fit the data you have.

• For photos and media, deduplication tools like rmlint or fdupes work at the filesystem level.
• For large block-level duplication across arrays, use zfs send/receive with checksums to validate transfers.
• For catalogue-driven systems such as Immich, export a manifest and compare hashes instead of trusting filenames.

Use checksums, not timestamps. SHA1 or xxhash are fine for large sets. Run a dry run first so you can see the candidates, then check them before removing anything. Keep a temporary quarantine area for 48–72 hours in case you delete something you still need.

Best practices for data handling

Keep the rules simple.

• One source of truth: decide which dataset is primary and mark the rest as copies.
• Archive policy: move rarely used data to a lower tier rather than keeping multiple hot copies.
• Snapshot hygiene: keep snapshot retention to what you can justify. Old snapshots are duplicate storage.
• Naming and metadata: keep folder layouts consistent so automation does not create accidental duplicates.

When I delete data, I move it first. Candidates go to a temporary hold on a single array for a week. If nothing complains, I remove them after that. It is slower than purge-and-pray, which is the point.

Reorganising storage arrays

Consolidation cut the mess down and made the lab easier to run. I moved from multiple R0x6 arrays to a single R5x10 array. That reduced management overhead and suited my workload better. It also let me retire an older VNX5300 and drop power consumption by roughly 75–100 W with a KTN-STL3 in the new setup.

When you reorganise:

• Plan capacity with growth in mind. Work from usable capacity after parity and replication.
• Map services to arrays by access pattern: fast arrays for VMs and active media, larger slower arrays for archives.
• Balance parity and drive count. More drives per RAID set improve capacity efficiency but rebuilds take longer. For my media, a single R5 array gave a decent mix of reliability and usable space.

If you are migrating, use rsync with checksums for file data or zfs send/receive for ZFS datasets. Test a restore from the new array before retiring the old one.

Network optimisation techniques

Networking decides how painful migrations and backup windows feel. I added a mix of 10 G and 2.5 G links. The 10 G link handles large dataset moves between servers. The 2.5 G links deal with daily traffic and media streaming.

Small changes help more than people like to admit:

• Jumbo frames can increase throughput for large file copies. Test with iperf3.
• Tune TCP window sizes for long transfers over the local LAN if latency is getting in the way.
• Use parallel streams for rsync or rclone to keep links busy during migrations.

If the storage array serves multiple clients, it needs enough CPU and network buffer to cope. If not, a small dedicated mover host takes the strain off the array.

Future considerations for backup systems

I care more about predictable restores than maximum redundancy. That meant keeping at least one off-site copy for disaster recovery, verifying backups regularly, and thinking about power costs before adding more arrays. Bigger drives reduce power per TB and chassis overhead.

I also ended up considering a move to Proxmox (PvE) because I wanted better networking and VM density than Hyper-V had given me before.

• Keep at least one off-site copy for disaster recovery.
• Automate verification. A backup that is never checked is a liability.
• Review power costs when adding arrays.
• Consider migration to Proxmox (PvE) if you need better networking or VM density than Hyper-V offered in my previous setup.

Think of backups as a flow: primary data, local replicate, off-site copy. Cut the amount of primary data by dealing with duplicates properly and the rest gets cheaper and quicker.

Final takeaways

• Audit first. Know where the duplicates live and how much space they waste.
• Use checksums and dry runs before deleting. Move before purge.
• Consolidate arrays where the reliability gains justify it. I went from five arrays to a plan centred on a single R5x10 array and backup arrays sized for about 10 TB.
• Tune the network for large transfers, but only upgrade when you can use the extra throughput.
• Set a recovery policy and verify backups regularly.

That is what saved me space, power, and a fair bit of backup faff.