No Space Left on Device: The Curious Case of the Missing Inotify Watchers

There’s a special kind of dread that creeps in when your Docker container cheerfully starts throwing errors like this:

bashCopyEditOSError: [Errno 28] No space left on device

Your first instinct is to reach for df -h. Disk usage looks fine. Next stop: du -sh /*. Still fine. But the logs keep piling up, Paperless-NGX keeps crashing, and you’re about one journalctl -xe away from starting your résumé.

Welcome to the wonderful world of inotify—the Linux kernel subsystem that lets you watch files and directories for changes, and the unsung hero behind half your modern event-driven workflows. If you're running anything remotely event-based—think Syncthing, Paperless-NGX, Dropbox, VSCode’s live file watching, or even Kubernetes configs—you’re probably leaning on inotify.

But like many things in Linux, inotify comes with limits. And when you hit them, things get weird.

Let’s dig into what’s happening, how to fix it, and how to make sure it doesn’t happen again.

🧠 What is Inotify?

inotify (short for inode notify) is a Linux kernel feature that allows user-space applications to monitor changes to the filesystem. It watches for things like file creations, deletions, modifications, and moves—essentially, it’s a publish/subscribe model for your files.

Under the hood, this works by maintaining a table of watches—these are kernel objects that point to inodes and describe the kind of events the system should alert your application about. Each watch costs memory. And like all good things, there’s a limit.

That’s where things get interesting.

🧨 The “No Space Left on Device” Error

This error doesn’t mean your SSD is full.

It means your process tried to create a new inotify watch, and the kernel replied with “Nope, we’re full.” The default value for fs.inotify.max_user_watches on most Linux distros is a modest 8192. That’s usually fine… unless you’re running apps that recursively watch thousands of directories (hi, Paperless-NGX 👋).

Let’s test it.

cat /proc/sys/fs/inotify/max_user_watches

If you’re getting weird errors, and this value is something like 8192, congratulations—you’ve just solved your first mystery of the Deep Filesystem State™.

⚙️ Raising the Limit

To raise the maximum number of inotify watchers per user persistently, you can add a line to your sysctl configuration:

echo "fs.inotify.max_user_watches=204800" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

This bumps the limit to 204,800, which is a reasonable ceiling for most use cases without blowing your RAM budget. After that, your apps should stop complaining.

🧪 Pro Tip: Want to test it on-the-fly, without persistence?

echo 204800 | sudo tee /proc/sys/fs/inotify/max_user_watches

That change works immediately but won’t survive a reboot.

📦 Special Case: Synology NAS

Now here’s a twist—if you’re running Paperless-NGX in Docker on a Synology NAS, welcome to BSD-on-Linux territory.

Synology’s underlying system won’t respect /etc/sysctl.conf. You can try, but it’s like writing in pencil on a whiteboard—gone with the next restart.

Instead, create a User Defined Script that runs at boot. Here’s how:

1. Open Control Panel → Task Scheduler
2. Create a Triggered Task → Boot-up
3. Under Run Command, enter: echo 204800 | sudo tee /proc/sys/fs/inotify/max_user_watches
   
4. Apply. Reboot. Celebrate.

This ensures the change takes effect on every boot, without manually hacking config files that Synology will gleefully overwrite.

🧩 Caveats, Gotchas, and Observations

• Each watch costs memory. You don’t get a free lunch. At 204k watchers, you're allocating around 32MB of kernel memory per process. It’s not a lot—unless you do this across dozens of containers.
• Too many watches ≠ better performance. Adding more watches doesn’t make things faster, it just lets you handle more files. If your app is already misbehaving with fewer, scaling up will just delay the failure.
• Docker images may have hardcoded limits. Especially minimal distros like Alpine. Make sure to check your container’s sysctl settings, especially if you use --privileged.
• Use watch -n 1 "cat /proc/sys/fs/inotify/max_user_watches" for live debugging. Combine with lsof | grep inotify to check open inotify file descriptors.

🔍 Diagnosing Inotify Usage

Want to see how many watchers are currently active?

find /proc/*/fd -ls 2>/dev/null | grep inotify

Or for total counts across processes:

lsof | grep inotify | wc -l

And to identify who is being greedy:

sudo find /proc/*/fd -lname anon_inode:inotify \
  | cut -d/ -f3 \
  | xargs -I{} sh -c 'echo PID {}: $(ps -p {} -o comm=)'

🪛 Real-World Example: Paperless-NGX

Paperless-NGX uses inotify extensively to monitor its input folders—great for responsiveness, but deadly if you’ve got a huge archive or lots of documents spread across nested directories.

You’ll see errors like:

OSError: [Errno 28] No space left on device

Don’t be fooled—your disks are fine. But Paperless is trying to watch too much. Increase the watcher count, and everything hums along again.

Also, consider tuning your watch paths to only what you really need monitored.

🚀 Wrapping Up

inotify is one of those Linux features that quietly does the heavy lifting—until it doesn’t. And when it fails, the symptoms are cryptic, the errors misleading, and the fixes obscure. But once you know where to look, it’s a two-minute fix with massive impact.

So next time your container cries “No space left on device,” don’t panic. Just smile knowingly and whisper:

“That’s not a disk error. That’s a watcher problem.”

🧠 TL;DR

• That OSError 28? Probably not disk space—it’s inotify watchers.
• Check current max: cat /proc/sys/fs/inotify/max_user_watches
• Raise it persistently:
  echo "fs.inotify.max_user_watches=204800" | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
• For Synology: run
  echo 204800 | sudo tee /proc/sys/fs/inotify/max_user_watches
  as a boot script, not via sysctl.conf.
• Monitor usage, avoid over-watching, and happy hacking!