AMD CPU SDPA result error

[DickyQi]

context | 编译/运行环境 | バックグラウンド

Ubuntu 24.04 AMD Ryzen9 CPU 9550x + 5090Dv2

how to reproduce | 复现步骤 | 再現方法

1. clone https://github.com/opencodewin/LocateAnything-ncnn
2. download models
3. build project with no vulkan support
4. run
5. text decoder result error

more | 其他 | その他

1. this project run with vulkan, text decoder result error
2. all test is run on fp16 mode

[nihui]

In LocateAnything-ncnn project, the KV cache trimming process modifies the ncnn::Mat objects representing the KV cache. Since the actual data layout of these Mat objects is backend-specific—and ncnn_llm merely passes them through during extraction and input—no issues have arisen so far.

For the AMD 9950X, which supports the AVX-512 BF16 instruction set, an interleaved KV cache layout is used to optimize AVX-512 BF16 dot-product efficiency. Similarly, GPU backends employ different layouts depending on specific hardware capabilities. More complex memory layouts may be implemented in the future to meet the requirements of high-performance kernels.

The layout conventions for KV cache Mat objects are currently unstable. Utility functions for manipulating these objects—such as clone and truncate—will be added in the future, but not right now.