vfs: fix AB-BA deadlock between dentry_hash_lock and the vnode lock

[gburd]

The VFS has an implicit lock order that namei() establishes and drele()
violated. This series states the rule, adds an assertion that enforces it, and
makes drele() conform.

The rule

fs/vfs/vfs_lookup.cc holds the directory vnode lock across the dentry cache
lookup, at two sites (:178 and :284):

vn_lock(dvp);
dp = dentry_lookup(mp, node);          // takes dentry_hash_lock
if (dp == nullptr) {
    error = VOP_LOOKUP(dvp, name, &vp);
    dp = dentry_alloc(ddp, vp, node);  // takes dentry_hash_lock
}
vn_unlock(dvp);

So the order is vnode lock, then dentry_hash_lock. Nothing may take a
vnode lock while holding dentry_hash_lock.

drele() did exactly that. It took dentry_hash_lock and then called
vn_del_name(), which takes the vnode lock internally
(fs/vfs/vfs_vnode.cc:531):

mutex_lock(&dentry_hash_lock);
if (--dp->d_refcnt) { ... }
LIST_REMOVE(dp, d_link);
vn_del_name(dp->d_vnode, dp);          // vn_lock(vp) in here
mutex_unlock(&dentry_hash_lock);

A namei() holding vn_lock and waiting for dentry_hash_lock, against a
drele() holding dentry_hash_lock and waiting for vn_lock, is an AB-BA
deadlock in which neither side can proceed.

Evidence

The first commit adds ASSERT(!vfs_dentry_hash_lock_held()) to vn_lock()
under DEBUG_VFS, so the rule is checked rather than asserted in prose. On
unfixed code it fires immediately, during vfs_init, with this stack:

0x00000000403bc1b8 <vn_lock+168>
0x00000000403bcc71 <vn_del_name+17>
0x00000000403c1167 <drele+103>
0x00000000403c0be7 <vfs_file::close()+87>
0x00000000403b8090 <fdrop+80>
0x00000000403b8216 <fdclose(int)+118>
0x00000000403b1267 <close+23>
0x00000000403b6832 <unpack_bootfs()+258>
0x00000000403b69b6 <vfs_init+182>

On fixed code, same test and same configuration, the assertion is silent and
the test passes:

VFSFLAG ... unset, default 1
Cmdline: /tests/tst-dentry-lock.so 8 40
tst-dentry-lock: 8 threads x 40 iters
tst-dentry-lock: 2880 ops, 0 failures
OK

Note what the stack shows: fdrop() is the ordinary close() path
(fs/vfs/main.cc:154, :159, :194). The illegal order is therefore taken by
every close() that drops a last dentry reference. It is not a rare
interleaving; the lock cycle is constructed constantly, and the only missing
ingredient for the deadlock is a concurrent namei() on the same vnode.

That ingredient was supplied in production. On a 32-vCPU guest running a
forking server with 184 concurrent backends, the system wedged with
dentry_hash_lock held, every vCPU halted in do_idle with an empty wakeup
mask, and over a thousand threads parked, while ~27 GiB of memory was free.

The included stress test passes on unfixed code. 48 threads x 400
iterations, 172,800 operations, 16 vCPUs under TCG, and it does not wedge:
OSv's mutex hold times here are short, and closing the cycle needs genuine
parallelism plus a same-vnode collision. A reviewer who runs it on a laptop and
sees it pass should not conclude the defect is fictional; the assertion above
is the reliable signal, and the hang itself was observed on 32 real vCPUs. The
test is included because it exercises the paths (concurrent open/close in one
directory, plus rename and unlink churn) and would catch a regression that
reintroduces a blocking call under the lock.

The fix

Release dentry_hash_lock before calling vn_del_name(). Unlinking from the
hash chain stays inside the critical section, and that is what makes the early
release safe: once the dentry is off the chain, dentry_lookup() cannot find
it, so no thread can take a new reference. The releasing thread is its sole
owner and the refcount is 0 and stays 0.

dentry_hash_lock becomes a leaf lock: it calls out to no other subsystem.
Every acquisition is routed through accessors so a new call site cannot quietly
bypass the rule, and the per-thread ownership tracking that the assertion needs
costs nothing when DEBUG_VFS is off.

Commits

1. vfs: fix AB-BA deadlock between dentry_hash_lock and the vnode lock - the
   ordering fix, the documented rule, and the assertion.
2. vfs: do not allocate while holding dentry_hash_lock in dentry_move() - an
   independent latent defect found while auditing the other call sites.
   dentry_move() called strdup() inside the critical section; that
   allocation can block in the page allocator while every lookup in the system
   waits. Hoisted above the lock. Not the deadlock, and on the rename() path
   rather than the open() path, but the same class of mistake.
3. tests: concurrency stress for the dentry cache lock order - the test,
   with the caveat above recorded in its header comment.

Notes

Not fork-specific. fork supplies the concurrency that closes the cycle, but
the inversion is present regardless: there are no CONF_fork references in
this series, and vfs_dentry.cc and vfs_vnode.cc compile clean with
conf_fork=0.

A separate change splitting the global dentry_hash_lock into per-bucket locks
is deliberately not in this PR. The 32 hash buckets have always shared one
mutex taken at 14 sites, which is a scalability defect, but it is independent of
this correctness fix and a reviewer may reasonably want one without the other.
It also has to come after this fix, because a per-bucket lock inverts against
the vnode lock exactly as the global one does. Its throughput effect is
currently unmeasured and no claim is made about it.

Tested on x86_64: 12 of 12 relevant tests pass (tst-vfs, tst-symlink 141
assertions, tst-remove 38, tst-pipe 87, tst-mmap, tst-condvar,
tst-yield, tst-dentry-lock, and three fork tests). tst-openat reports one
pre-existing failure that reproduces identically on unpatched master.

[gburd]

End-to-end confirmation on a real workload, in addition to the lock-order assertion already in the PR.

A PostgreSQL-on-ZFS HammerDB run on a base WITHOUT this fix wedged the guest hard at 32 virtual users: top -H on the host showed all 35 qemu vCPU threads asleep, zero runnable, with ~27 GiB free. That is the AB-BA deadlock this PR fixes-a true all-asleep hang, not a slow path (a lost wake or memory stall would make progress with that much memory free).

Rebuilding the identical image with this fix applied and re-running the same 32-VU cells: no wedge, three cells completed at ~62,400 NOPM, with ~18 of 32 vCPUs busy (vs 0 in the wedge). So the fix removes the deadlock end-to-end under a real database workload, not just under the lock-order assertion.

To be clear about scope: clearing the deadlock lets the workload run at 32 VU, but throughput there is still well below Linux on the same hardware-that ceiling is a separate WAL-commit-completion serialization issue, not this deadlock, and is not something this PR claims to address. This PR's claim is narrower and now has both forms of proof: the lock-order assertion fires on the unfixed path, and the workload that deadlocked without it completes with it.