Object wrapping and debug utils callbacks #6972

Describe the Issue

The unique handle wrapping feature results in VVL replacing object handles returned by layers/ICD with VVL's own handles. VVL then handles the wrapping/unwrapping to ensure that this is all hidden from the application.

However, it does not perform this translation on handles passed to the debug utils callback via VkDebugUtilsMessengerCallbackDataEXT::pObjects[0]::objectHandle. A specific case where this is visible is when using VK_EXT_device_address_binding_report, which uses the debug callbacks to return memory binding messages.

I believe the sequence of events here is:

1. Application creates a debug messenger, and passes a callback function pointer
2. VVL sees that the ICD implements VK_EXT_debug_utils (which is a required dependency of VK_EXT_device_address_binding_report) and so defers to the ICD to create the debug messenger
3. The application calls an object creation function and VVL returns a wrapped handle
4. The ICD fires a memory binding message via the debug callback, but passes it's own handles in VkDebugUtilsMessengerCallbackDataEXT::pObjects[0]::objectHandle. The debug callback that gets invoked is the applications own function pointer, and so VVL doesn't get to intervene/wrap the handles to match those returned by step 3. The application now has no way to correlate the two handles.

It seems like VVL should be able to intercept the creation of the debug messenger, and pass it's own version of the debug messenger callback down to the ICD/lower layers. VVL's callback would need to perform wrapping/unwrapping and then invoke the applications own callback.

I haven't actually tried to implement this, so there may well be corner cases that make this not practical. If that's the case, then the fallback solution would likely be to add a note to the spec/extension documentation recommending disabling object wrapping in VVL when using VK_EXT_device_address_binding_report. However, since both the extension and VVL are useful debugging tools, it would be preferable to find a route to making them work together.

Expected behavior

From the applications PoV: handles returned via the debug callbacks should be consistent with those returned via the object creation functions.

Additional context

Whilst the motivating use case for this is VK_EXT_device_address_binding_report, I think you can trigger the same behaviour by having a 2nd layer located underneath VVL, which then calls vkSubmitDebugUtilsMessageEXT after the application has created a debug messenger.