Replacing iwatch: 20,000 mails, an XML file, and a weekend of Rust

What this is about Link to heading

My server has been running iwatch for years. It watches directories with inotify and reports changes by mail. The tool does what it promises — it just does it in a way that became unusable over time.

Two things pushed me to replace it.

One mail per event. A kernel update touches thousands of files. The mailbox gets thousands of mails, all within seconds. Whatever was actually suspicious in between is no longer findable.

The configuration. An excerpt from my old iwatch.xml:

<path type="recursive">/dev</path>
<path type="exception">/dev/shm/libpod_rootless_lock_5001</path>
<path type="exception">/dev/shm/libpod_rootless_lock_1002</path>
<path type="regexception">.*PostgreSQL.*</path>

That looks more harmless than it is. exception is a literal path, regexception is an unanchored regular expression. The latter visually sits under /dev but applies to the entire system. And because wildcards only exist in the regex variant, I ended up listing the Podman locks one UID at a time — instead of writing libpod_rootless_lock_* once.

At some point I stopped maintaining the file. That is exactly the problem: a monitoring tool whose configuration nobody touches eventually watches the wrong things.

The two decisions Link to heading

Group instead of one by one. Events are collected in a time window — ten seconds in my case — and sent as one mail. 20,000 mails become a handful.

Two lists instead of one. include and exclude, neither of them order-dependent, exclude always wins:

include = ["/var/lib/roundcube/**"]
exclude = ["/var/lib/roundcube/logs/**"]

I was convinced for a long time that I needed three levels — include, exclude from that, include again from that. When I went through my old XML rule by rule, not a single one was three-level. Every one of them was: watch tree X, but not thing Y inside it. The feeling of nesting did not come from the requirement, it came from having two different kinds of exception.

A side effect: anything not covered by include is never registered with the kernel in the first place. The old file listed /proc as recursive and then threw all of it away via regexception — tens of thousands of watches for nothing.

What modify has to do with the noise Link to heading

The single largest source of noise is IN_MODIFY. It fires on every write. Copying a 200 MB file produces thousands of events for one file.

IN_CLOSE_WRITE fires exactly once: when the file has been written and closed. That is almost always what you actually want to know.

Hence the default set:

events = ["create", "close_write", "attrib", "delete", "moved"]

modify can be switched on by anyone who needs it — for log files, say, which a process keeps open and which therefore never emit a close_write.

Worth remembering: anything not listed in events is filtered by the kernel itself. It never reaches the daemon and costs nothing.

Reading is harmless, by the way: cat produces IN_ACCESS and IN_CLOSE_NOWRITE, not IN_CLOSE_WRITE.

Rust, without knowing Rust Link to heading

I had never written Rust. Java, C, and thirty years of Linux, but no Rust. The rebuild was a good excuse.

Two things helped more than the language itself.

Test-driven, with time passed in from outside. The core — filtering, grouping, building the mail text — knows nothing about the kernel, the clock, or the mail system. Instead of reading the time inside the code, it is passed as a parameter:

pub fn is_due(&self, now: Instant, window: Duration) -> bool {
    !self.is_empty() && now.duration_since(self.window_start) >= window
}

That makes “what happens after ten seconds” testable without waiting ten seconds. The whole test suite runs in milliseconds — no real files, no mail, no waiting.

enum plus match. When I later added two event kinds, the compiler pointed me at exactly the place where the handling was missing. No forgotten case, no silent fall-through.

The ownership rules were, as expected, where I fought the compiler most. The error messages are good enough that it helps more than it hurts while learning.

Four things that only surfaced on review Link to heading

After the first working version I went through the project again, this time looking for security problems. That paid off.

Symlinks. I had used entry.metadata() to detect directories. That follows the symlink. A link pointing at / would have pulled the entire filesystem into the watch set. The correct call is entry.file_type(), which does not follow.

Silent failures. When a watch could not be registered — because max_user_watches was reached, for example — the directory was skipped without comment. The daemon would then have run with half the coverage while looking perfectly healthy. For a security tool that is the worst variant: it reports nothing, and you take that as a good sign.

Unbounded memory. If the mail system is down, the daemon keeps collecting. There is now a max_events; once it is reached the mail goes out immediately.

Header injection. The host name ends up in the Subject. File names on Linux may contain almost anything, including line breaks — and /tmp is in my include. Without sanitising, anyone who can write there could smuggle in their own headers, a Bcc: for instance. The test for it was wrong at first: I checked whether the string "Bcc:" appeared, instead of whether an extra header line had been created.

What the recipient cannot see, they cannot judge. So incomplete reports — kernel overflow or max_events — now carry INCOMPLETE in the subject line.

What it looks like Link to heading

[fwatchd] server1: 3 changes

3 changes on server1

2026-09-12 14:02:11  modified   /etc/passwd
2026-09-12 14:02:14  created    /etc/hosts
2026-09-12 14:02:19  deleted    /etc/fstab

The configuration can be checked before anything goes live:

fwatchd check /etc/fwatchd/fwatchd.toml /etc/passwd /proc/1/status

WATCHED  /etc/passwd
ignored  /proc/1/status

This is the part I missed most with iwatch. My old XML contained an exception with a stray quotation mark at the end of the line. It probably never worked, and I only noticed while translating it. Unknown keys now abort at startup instead of being silently ignored.

What is still open Link to heading

It still relies on inotify, which means one watch per directory — roughly one kilobyte of kernel memory that cannot be swapped out. fanotify would attach to a whole filesystem instead, but it needs CAP_SYS_ADMIN and has little usable library support in Rust. The event source is therefore built to be swappable; if it becomes necessary, it comes in as a second adapter.

Overflow remains an issue either way. It cannot be defined away — only reported honestly.

Source and packages: codeberg.org/randomDuderus/fwatchd, MIT licensed. Questions, corrections and your own experiences are welcome in the comments.