Reversing the flow: from object storage back to the NAS

What this is about Link to heading

A rented server backs itself up to object storage every night using restic. At home, the NAS pulls a second copy of that repository every Sunday. The direction was a deliberate choice: the NAS pulls, so no port needs to be open at home.

That setup has failed — not because of a mistake, but because of a calculation nobody ever made. Rebuilding it then surfaced two configuration errors that had been invisible until that point. One of them is the real reason for this post.

The trigger Link to heading

An email from Backblaze at 06:03: Download Bandwidth Cap Reached 100%. Not for the first time.

That is worse than it sounds. Once the download cap is reached, B2 answers every download with a 403. And restic reads the config file in the repository before every operation. A tripped read counter therefore blocks writing as well. For a whole day not a single operation against the repository was possible — no snapshots, no stats, no backup.

Takeaway: Couple egress to availability and you haven’t built a brake, you have built a cascade.

The diagnosis Link to heading

Step by step, and deliberately using methods that cost no bytes themselves:

1. The nightly backup completed cleanly at 03:08, in nine seconds. So the cap was tripped after the backup, not by it.
2. No second consumer on the server. No prune timer, no cron entry, only the one systemd unit references restic.
3. The cache was warm and had been written to continuously for weeks. Nothing is being re-fetched unnecessarily.
4. The snapshot sizes in the local log showed a jump of +417 MiB per day over several days, then flat. The cause was freshly compressed PostgreSQL dumps every day. Every compression run produces different bytes, so restic cannot deduplicate anything. Switching to uncompressed dumps removed the effect — but permanently raised the repository’s baseline size.
5. Live test: restic stats ran into a b2_download_file_by_name: 403. The cap was tripped right then, not reconstructed after the fact.
6. In the object storage panel: three application keys. Alongside the master key and the server’s write key, a third one with read permissions only. The permission profile of a pulling consumer.
7. On the NAS: the directory holding the second copy, with timestamps in data/ all on Sundays, all at exactly 06:00. The email arrived at 06:03.
8. The script: rclone copy with --transfers 8 and no --bwlimit. Eight parallel streams on a full line pull a gigabyte in three minutes.

The flawed assumption Link to heading

Two numbers from the same free tier had been mixed up:

They sit directly below one another in the documentation. Mixing them up matters, because it hides a structural impossibility:

A 2.6 GB repository can never be read in full in a single day on a 1 GB daily budget. Any pulling second copy is doomed the moment the repository grows past the daily limit.

At first this went unnoticed, because only the weekly delta was fetched and that was small enough. Since the dumps sit uncompressed in the repository, the delta has grown past the limit. From that point on the job structurally cannot get through, no matter how often it tries.

There is a second point the error message doesn’t touch at all: a NAS that pulls from object storage pays for every byte twice. Once on the way up from the server, once on the way back down. And the source data is sitting on the server locally anyway.

The decision Link to heading

The download budget stays at zero — no cost, not even potential cost. The price is that every operation has to stay below one gigabyte.

The fix is not a bigger budget but reversing the direction: the server pushes straight to the NAS. Object storage becomes a write-only target, and the download cap is never touched again.

The downside deserves to be named. A pulling NAS has no write access to the source: if the server is compromised, the second copy is safe. With a push it is the other way round. The remedy is --append-only in rest-server: the server may write but not delete. That is enabled — and it is exactly why retention for this repository lives on the NAS rather than on the server. Whoever may not delete cannot prune either.

Takeaway: Pull protects the source, push needs --append-only. Both are legitimate models. You just have to know which one you picked, and why.

The same three participants, one direction reversed — and an impossibility turns into 26 seconds.

The rebuild Link to heading

The problem first: the NAS runs DSM on kernel 4.4, which knows nothing about WireGuard — that arrived with 5.6. A container cannot load a kernel module that doesn’t exist. So the direct route via the Synology was out.

The solution was closer to hand than it first looked: the OPNsense box has been a WireGuard endpoint since Part 11. The rented server simply becomes another peer. That preserves the original property of the setup — no additional port open at home.

Three details worth stating:

• The tunnel address 10.0.60.99, deliberately far above the existing client addresses. The gap makes it visible at a glance that this is not an ordinary roaming client.
• A second interface wg1 on the server. Its wg0 is a separate, independent instance serving a different purpose. The two must not be mixed.
• AllowedIPs on the server side is the NAS address only, not 0.0.0.0/0. A full tunnel would have pushed all of the rented server’s traffic through my home network and taken down every service running on it. The double meaning of that field was already a topic in Part 11.

I decided against a second WireGuard instance. It would have been “more separated”, but it would have cost a second open UDP port on the WAN side. Here the separation is done by the tunnel address plus a firewall rule. In this network permissions are granted per address anyway, not per instance.

The VLAN move in the middle of it Link to heading

Halfway through the rebuild I noticed the NAS was still sitting in 10.0.1.0/24 — together with the firewall, the switch and the virtualisation host. In other words, in the middle of the management network.

A NAS does not belong there. It is a device with many open ports, a web interface and a package system you cannot fully oversee. Putting exactly that next to the firewall devalues the rest of the segmentation.

So: new VLAN 32, 10.0.32.0/24. On OpenWrt (DSA, bridge-vlan): create VLAN 32 with sfp-wan:t and lan2:u*, then remove lan2 from VLAN 10. Which port was the right one could be determined from the MAC table — the OUI 90:09:D0 belongs to Synology.

The firewall rules are the four-rule chain from Part 10, one to one: DNS to the firewall, NTP, block on RFC1918, then internet. The block rule is the whole point. If the NAS is compromised, it reaches no other internal network.

What went wrong Link to heading

The sync that had stopped Link to heading

This is the better of the two findings. After the VLAN change it turned out that the Synology Drive client on the workstation still had a 192.168.x.x address configured as its target — wrong since an earlier rebuild, and dead ever since.

So the client synchronised nothing. Not a little, not slowly: nothing. And it never said so. No error, no warning, no red icon, no email. The interface looked exactly as it always had.

It only came to light when I entered the correct address after the move into the new VLAN. At that point it started catching up on the entire backlog — and only the size of that backlog showed how long this had been going on.

The interface was unremarkable the whole time. The numbers were not.

This is exactly the same pattern as the dynamic DNS script from Part 11. That kept running too, it just didn’t deliver anything.

Takeaway: A service that doesn’t complain is no proof that it is working. The most dangerous failure state is the one that looks like normal operation.

In practice that means: after every rebuild, don’t check whether something runs, check what it depends on and when it last did anything. The configured address, the address actually assigned, the timestamp of the last transfer.

The rule below the block rule Link to heading

The second finding is the classic one. The firewall rule for the rest-server port sat below the block rule on RFC1918 — and the NAS address is a private address. So it was blocked. The same mistake as with IPTV in Part 10.

Diagnosed with wg show: roughly 2.9 KiB sent, 624 B received. That asymmetry says everything — packets go out, nothing comes back, so the chain breaks behind the tunnel. The firewall’s live log then showed the blocked lines in plain text. The “check from the outside in” method from Part 11 proved itself for the second time.

The unexpected bonus Link to heading

Comparing the two repositories turned up something unexpected. The old copy on the NAS is a superset of the object storage repository:

The reason: the script used copy, not sync. Deletions were deliberately not mirrored. So the copy still holds the August snapshots that a prune removed from the source repository long ago. A comment in the script says exactly that — an old decision that paid off unexpectedly.

And because the copy is local, a restic check --read-data was possible for the first time: 293 packs, 77 seconds, no errors. That is the first verification of this repository’s actual contents, ever. Against object storage with an exhausted daily budget, that check simply cannot be run.

One more number, from the next day’s fresh budget window: prune would have freed 336 MiB in this repository, a good one percent of its total size. Effectively a no-op. The snapshots that forget discards every day share almost all of their blobs with the ones that remain. The worry that “without a timer the repository grows silently” does not apply here.

Where I stand now Link to heading

rest-server runs as a container on the NAS, reachable only through the tunnel and only from the one server address. The NAS sits in VLAN 32 and reaches no other internal network from there.

The first run: 3.497 GiB in 26 seconds. For comparison: fetching the same amount from object storage at one gigabyte per day would have taken three and a half days. That is the difference between push and pull in a single number.

restic check: no errors were found.

The NAS repository was deliberately initialised fresh rather than reusing the old copy. Two repositories with identical IDs would share the same cache on the server — the kind of mistake you only discover during a restore.

The run runs on its own Link to heading

On the server a timer fires daily at 04:00, with a quarter hour of jitter and Persistent=true so a missed run is caught up. The unit carries a Requires=wg-quick@wg1.service. Without the tunnel there is no run — rather than half a run.

Three things in the script behind it are worth mentioning:

Its own dead man’s switch. The NAS run pings its own check: /start at the beginning, /fail via a trap … ERR, the success ping at the end. That was the whole point. A NAS that happens to be switched off now reports this one check as overdue and leaves the backup to object storage untouched.

Its own consistent database snapshot via sqlite3 .backup, independent of the other backup. The live file itself is excluded — copying a SQLite database while it is in use produces a file that looks like a backup and isn’t one.

A timeout 2h around the restic run. A hung run would otherwise block the next one and still report nothing while doing so.

Cleanup happens on the NAS itself. A container there runs forget with --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune on the first of every month at 05:00. Retention therefore sits with whoever holds the data, not with whoever sends it.

Next step Link to heading

One item is still open, and it belongs here rather than in a drawer: the old pull copy still sits next to the new repository under its old name. It holds 32 snapshots with the August states that exist nowhere else. It needs to be renamed and frozen as an archive so that it doesn’t get swept up in some cleanup one day — of all copies, the one that holds something no longer reproducible.

After that the setup is complete: one copy off site, one in the house, both monitored, both verified.

Written in September 2026.