many new tests in 2.15 fail when testing on kernel 6.12

[jirislaby]

Like this:

[  170s] Running test cancel-fd-userdata.t                                   cancel failed: -2
[  170s] test normal fd failed
[  170s] Test cancel-fd-userdata.t failed with ret 1

[  170s] Running test file-alloc-range-hint.t                                alloc returned slot 2, expected [10, 20)
[  170s] test_hint_below_range failed
[  170s] Test file-alloc-range-hint.t failed with ret 1

[  170s] Running test nop-fixed-file-leak.t                                  BUG: write to pipe with closed read end succeeded -- NOP leaked a file reference
[  170s] Test nop-fixed-file-leak.t failed with ret 1

[  170s] Running test poll-mshot-wake.t                                      IORING_CQE_F_MORE set
[  170s] Test poll-mshot-wake.t failed with ret 1

[  170s] Running test read-inc-buf-more.t                                    BUF_MORE not set, res=32 flags=0x1
[  170s] file read 0 failed
[  170s] test_file failed
[  170s] Test read-inc-buf-more.t failed with ret 1

[  170s] Running test recv-bundle-buf-len.t                                  case2: expected res=32, got 64
[  170s] test_success_trim failed
[  170s] Test recv-bundle-buf-len.t failed with ret 1

[  170s] Tests failed (6): <cancel-fd-userdata.t> <file-alloc-range-hint.t> <nop-fixed-file-leak.t> <poll-mshot-wake.t> <read-inc-buf-more.t> <recv-bundle-buf-len.t>

[axboe]

Stable is quite behind these days... Check again in a 6.12-stable release or two and it should look better I suspect. Which version are you on now, you didn't mention that.

[jirislaby]

Which version are you on now, you didn't mention that.

Oops, sorry. This is SUSE Linux 16.1. It's not following stable 1:1. I am not sure you desire to support distros. We currently do:

%if 0%{?suse_version} < 1600
TEST_EXCLUDE+=( accept-non-empty.t )
TEST_EXCLUDE+=( accept.t )
TEST_EXCLUDE+=( bind-listen.t )
TEST_EXCLUDE+=( cancel-fd-userdata.t )
TEST_EXCLUDE+=( eploop.t )
TEST_EXCLUDE+=( eventfd-ring.t )
TEST_EXCLUDE+=( evloop.t )
TEST_EXCLUDE+=( fallocate.t )
TEST_EXCLUDE+=( fd-pass.t )
TEST_EXCLUDE+=( fifo-futex-poll.t )
TEST_EXCLUDE+=( file-alloc-range-hint.t )
TEST_EXCLUDE+=( fixed-buf-merge.t )
TEST_EXCLUDE+=( fixed-seg.t )
TEST_EXCLUDE+=( fpos.t )
TEST_EXCLUDE+=( link-timeout.t )
TEST_EXCLUDE+=( min-timeout.t )
TEST_EXCLUDE+=( min-timeout-wait.t )
TEST_EXCLUDE+=( nop.t )
TEST_EXCLUDE+=( poll-mshot-wake.t )
TEST_EXCLUDE+=( read-before-exit.t )
TEST_EXCLUDE+=( read-inc-buf-more.t )
TEST_EXCLUDE+=( recv-bundle-buf-len.t )
TEST_EXCLUDE+=( recv-msgall-stream.t )
TEST_EXCLUDE+=( recv-mshot-fair.t )
TEST_EXCLUDE+=( recvsend_bundle.t )
TEST_EXCLUDE+=( register-restrictions.t )
TEST_EXCLUDE+=( rsrc_tags.t )
TEST_EXCLUDE+=( send_recv.t )
TEST_EXCLUDE+=( sendzc-bug.t )
TEST_EXCLUDE+=( socket-rw-offset.t )
TEST_EXCLUDE+=( sqwait.t )
TEST_EXCLUDE+=( sync-cancel.t )
TEST_EXCLUDE+=( timeout.t )
TEST_EXCLUDE+=( wq-aff.t )
%endif
%if 0%{?suse_version} >= 1600 && %{suse_version} < 1699
TEST_EXCLUDE+=( cancel-fd-userdata.t )
TEST_EXCLUDE+=( file-alloc-range-hint.t )
TEST_EXCLUDE+=( min-timeout.t )
TEST_EXCLUDE+=( min-timeout-wait.t )
TEST_EXCLUDE+=( nop-fixed-file-leak.t )
TEST_EXCLUDE+=( poll-mshot-wake.t )
TEST_EXCLUDE+=( read-inc-buf-more.t )
TEST_EXCLUDE+=( recv-bundle-buf-len.t )
%endif
%if 0%{?suse_version} >= 1600
TEST_EXCLUDE+=( io-wq-exit.t )
%endif

BTW there are a couple of new failures against 7.2-rc (not sure which of those are 2.15-release related):

[  211s] Running test iowait.t                                               iowait diff too small: 1
[  211s] Running test read-before-exit.t                                     queue_init: Cannot allocate memory
[  211s] Test IOPOLL failed loop 1
[  211s] Test read-before-exit.t failed with ret 1
[  211s] Running test read-inc-buf-more.t                                    ring setup failed: -12
[  211s] test_pipe failed
[  211s] Test read-inc-buf-more.t failed with ret 1
[  211s] Running test read-inc-file.t                                        ring setup failed: -12
[  211s] Test read-inc-file.t failed with ret 1
[  211s] Running test read-mshot.t                                           ring setup failed: -12
[  211s] test 0 0 0 failed
[  211s] Test read-mshot.t failed with ret 1
[  211s] Tests timed out (1): <multicqes_drain.t>
[  211s] Test run complete, kernel: 7.2.0-rc1-3.g0cbd3ff-default #1 SMP PREEMPT_DYNAMIC Sat Jul 4 00:09:47 UTC 2026 (0cbd3ff)
[  211s] Tests failed (5): <iowait.t> <read-before-exit.t> <read-inc-buf-more.t> <read-inc-file.t> <read-mshot.t>

But they point to ENOMEM?

[jirislaby]

This one fails with 7.1.2 too:

[  188s] Running test nop-fixed-file-leak.t                                  BUG: write to pipe with closed read end succeeded -- NOP leaked a file reference
[  188s] Test nop-fixed-file-leak.t failed with ret 1

[ammarfaizi2]

BTW there are a couple of new failures against 7.2-rc

I can confirm iowait.t also fails on my testing against 7.2.0-rc2. This is the only test that fails.

# make -j128 runtests-parallel
make[1]: Entering directory '/root/liburing/src'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/liburing/src'
make[1]: Entering directory '/root/liburing/test'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/liburing/test'
make[1]: Entering directory '/root/liburing/examples'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/liburing/examples'
make[1]: Entering directory '/root/liburing/test'
Running test iowait.t                                               iowait diff too small: 0
test 1 failed
Test iowait.t failed with ret 1
Test run complete, kernel: 7.2.0-rc2-orinc-dbg-2026-07-07-gaff4cf2cf14b #2 SMP PREEMPT_DYNAMIC Tue Jul  7 07:10:59 UTC 2026
Tests failed (1): <iowait.t>
make[1]: *** [Makefile:423: iowait.run_test] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/root/liburing/test'
make: *** [Makefile:28: runtests-parallel] Error 2

@jirislaby's tests other than iowait.t against 7.2.0-rc1 seem to be running out of memory (<read-before-exit.t> <read-inc-buf-more.t> <read-inc-file.t> <read-mshot.t>).

-12 = -ENOMEM

[krisman]

Which version are you on now, you didn't mention that.

Oops, sorry. This is SUSE Linux 16.1. It's not following stable 1:1. I am not sure you desire to support distros. We currently do:

SUSE kernels don't follow stable as you know, and I'd claim it is our (SUSE) problem to make these tests pass. We also cherry-picked many features on top of 6.12, which are not supposed to appear in 6.12-stable. That is to say this should have been reported against the distro, not upstream. Have you seen bsc#1261250 for 16.1?

%if 0%{?suse_version} < 1600
TEST_EXCLUDE+=( accept-non-empty.t )
TEST_EXCLUDE+=( accept.t )
TEST_EXCLUDE+=( bind-listen.t )
TEST_EXCLUDE+=( cancel-fd-userdata.t )
TEST_EXCLUDE+=( eploop.t )
TEST_EXCLUDE+=( eventfd-ring.t )
TEST_EXCLUDE+=( evloop.t )
TEST_EXCLUDE+=( fallocate.t )
TEST_EXCLUDE+=( fd-pass.t )
TEST_EXCLUDE+=( fifo-futex-poll.t )
TEST_EXCLUDE+=( file-alloc-range-hint.t )
TEST_EXCLUDE+=( fixed-buf-merge.t )
TEST_EXCLUDE+=( fixed-seg.t )
TEST_EXCLUDE+=( fpos.t )
TEST_EXCLUDE+=( link-timeout.t )
TEST_EXCLUDE+=( min-timeout.t )
TEST_EXCLUDE+=( min-timeout-wait.t )
TEST_EXCLUDE+=( nop.t )
TEST_EXCLUDE+=( poll-mshot-wake.t )
TEST_EXCLUDE+=( read-before-exit.t )
TEST_EXCLUDE+=( read-inc-buf-more.t )
TEST_EXCLUDE+=( recv-bundle-buf-len.t )
TEST_EXCLUDE+=( recv-msgall-stream.t )
TEST_EXCLUDE+=( recv-mshot-fair.t )
TEST_EXCLUDE+=( recvsend_bundle.t )
TEST_EXCLUDE+=( register-restrictions.t )
TEST_EXCLUDE+=( rsrc_tags.t )
TEST_EXCLUDE+=( send_recv.t )
TEST_EXCLUDE+=( sendzc-bug.t )
TEST_EXCLUDE+=( socket-rw-offset.t )
TEST_EXCLUDE+=( sqwait.t )
TEST_EXCLUDE+=( sync-cancel.t )
TEST_EXCLUDE+=( timeout.t )
TEST_EXCLUDE+=( wq-aff.t )
%endif

Most of these are cases of the feature not being present in <SL-16.0 and the test not being able to catch that. If you run the testsuite of the liburing version shipped in SLE15-SP7, you'll see the list is much smaller. For instance, wq-aff, bind-listen, accept, register-restrictions, recv* all test stuff too new for SLE15-SP6 kernels that just throw -EINVAL or -EOPNOTSUPP immediately, thus we disabled the test. Other stuff like eploop were real bugs, fixed years ago (i.e. bsc#1207100), disabling the test is just wrong.

Yes, it would be great if the missing feature test skips, but the way we check for features in io_uring don't make it easy for the Frankestein kernels from distros. Still, by excluding them immediately in Factory without proper review, we risk not checking for real regression. can you report new failures as bugs instead of adding to the exclusion list?

%if 0%{?suse_version} >= 1600 && %{suse_version} < 1699
TEST_EXCLUDE+=( cancel-fd-userdata.t )
TEST_EXCLUDE+=( file-alloc-range-hint.t )
TEST_EXCLUDE+=( min-timeout.t )
TEST_EXCLUDE+=( min-timeout-wait.t )
TEST_EXCLUDE+=( nop-fixed-file-leak.t )
TEST_EXCLUDE+=( poll-mshot-wake.t )
TEST_EXCLUDE+=( read-inc-buf-more.t )
TEST_EXCLUDE+=( recv-bundle-buf-len.t )

Some of these actually surprise me, as 16.0 should be close to 16.1, so they don't fail there? there is a bug for the few 16.1 failures, and the 16.0 kernel used to pass fine, alas the spec in SLE.

%endif
%if 0%{?suse_version} >= 1600
TEST_EXCLUDE+=( io-wq-exit.t )
%endif

bsc#1261250

[krisman]

Fwiw, from your 16.0 list:

TEST_EXCLUDE+=( cancel-fd-userdata.t )
TEST_EXCLUDE+=( file-alloc-range-hint.t )
TEST_EXCLUDE+=( nop-fixed-file-leak.t )

These are new tests, missing kernel patch backports. Instead of blacklisting, we should have backported. I queued them all.

TEST_EXCLUDE+=( min-timeout.t )
TEST_EXCLUDE+=( min-timeout-wait.t )

Can't reproduce. Perhaps the OBS constraint environment are required to trigger it? Will look further why you see failures on them

TEST_EXCLUDE+=( read-inc-buf-more.t )
TEST_EXCLUDE+=( recv-bundle-buf-len.t )

Likely due to the partial backport of bundles. One of the ugly things we have in 6.12. I need to check but being discussed at 1261250

TEST_EXCLUDE+=( poll-mshot-wake.t )

Seems like a real failure. gotta check under bsc#1261250

[krisman]

Fwiw, from your 16.0 list:

TEST_EXCLUDE+=( cancel-fd-userdata.t )
TEST_EXCLUDE+=( file-alloc-range-hint.t )
TEST_EXCLUDE+=( nop-fixed-file-leak.t )

These are new tests, missing kernel patch backports. Instead of blacklisting, we should have backported. I queued them all.

TEST_EXCLUDE+=( min-timeout.t )
TEST_EXCLUDE+=( min-timeout-wait.t )

Can't reproduce. Perhaps the OBS constraint environment are required to trigger it? Will look further why you see failures on them

OK, these were only issues in sle-16.0. Fixed.

TEST_EXCLUDE+=( read-inc-buf-more.t )
TEST_EXCLUDE+=( recv-bundle-buf-len.t )

Likely due to the partial backport of bundles. One of the ugly things we have in 6.12. I need to check but being discussed at 1261250

Fixed as well. issue with prior backport.

TEST_EXCLUDE+=( poll-mshot-wake.t )

Seems like a real failure. gotta check under bsc#1261250

And fixed. Please test with the 16.0 and 16.1 kernels and we can unexclude them from the package!

[krisman]

To the upstream issues...

I can confirm iowait.t also fails on my testing against 7.2.0-rc2. This is the only test that fails.

This seems to be a racy test, though I don't get how it could be racy. I'm unable to reproduce it. It passes 100% of the time on system:

Running test iowait.t 2 sec
Test run complete, kernel: 7.2.0-rc2-linux@0e35b9b6ec0f-8a3ef #28 SMP PREEMPT_DYNAMIC Thu Jul 9 00:37:21 CEST 2026
All tests passed

Even more weird that your diff is 0. Could it be your test wasn't bound to cpu 0? Could share the output of

 $ sudo trace-cmd record -e 'io_uring*' ./iowait.t ; 
 $ sudo trace-cmd report

For this one:

This one fails with 7.1.2 too:

[  188s] Running test nop-fixed-file-leak.t                                  BUG: write to pipe with closed read end succeeded -- NOP leaked a file reference
[  188s] Test nop-fixed-file-leak.t failed with ret 1

Stable is just missing upstream commit 2564ca2e31bd ("io_uring/nop: fix file reference leak with IOSQE_FIXED_FILE"). It should flow there through the AUTOSEL, since it has a fixes tag and stable was CC'ed.

[jirislaby]

Even more weird that your diff is 0. Could it be your test wasn't bound to cpu 0? Could share the output of
$ sudo trace-cmd record -e 'io_uring*' ./iowait.t ;
$ sudo trace-cmd report

cpus=160
        iowait.t-2093  [000] .....    49.754770: io_uring_create:      ring 0xffff8f27d317c800, fd 3 sq size 64, cq size 128, flags 0x10000
        iowait.t-2093  [000] .....    49.754816: io_uring_submit_req:  ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, opcode READ, flags 0x400000, sq_thread 0
        iowait.t-2093  [000] .....    49.754818: io_uring_file_get:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, fd 4
        iowait.t-2093  [000] .....    49.754827: io_uring_poll_arm:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, opcode READ, mask 0xc000004b, events 0xc000207b
        iowait.t-2093  [000] .....    49.755487: io_uring_cqring_wait: ring 0xffff8f27d317c800, min_events 1
        iowait.t-2093  [000] d..1.    50.756042: io_uring_task_add:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, opcode READ, mask 0
        iowait.t-2093  [000] ...1.    50.756049: io_uring_complete:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, result 0, cflags 0x0 extra1 0 extra2 0 
        iowait.t-2093  [000] .....    50.756051: io_uring_task_work_run: tctx 0xffff8f27cccbcc00, count 1
        iowait.t-2093  [000] .....    50.756062: io_uring_submit_req:  ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, opcode READ, flags 0x400000, sq_thread 0
        iowait.t-2093  [000] .....    50.756063: io_uring_file_get:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, fd 4
        iowait.t-2093  [000] .....    50.756064: io_uring_poll_arm:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, opcode READ, mask 0xc000004b, events 0xc000207b
        iowait.t-2093  [000] .....    50.756335: io_uring_cqring_wait: ring 0xffff8f27d317c800, min_events 1
        iowait.t-2093  [000] d..1.    51.756874: io_uring_task_add:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, opcode READ, mask 0
        iowait.t-2093  [000] ...1.    51.756877: io_uring_complete:    ring 0xffff8f27d317c800, req 0xffff8f27e048f600, user_data 0x0, result 0, cflags 0x0 extra1 0 extra2 0 
        iowait.t-2093  [000] .....    51.756878: io_uring_task_work_run: tctx 0xffff8f27cccbcc00, count 1

It fails with rather low frequency:

~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
iowait diff too small: 12
test 1 failed
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
iowait diff too small: 43
test 1 failed
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test # ./iowait.t                   
iowait diff too small: 29
test 1 failed

The diff is varying, sometimes it is 0.

[krisman]

Ok. got it. the issue can happen if you have noise in the same cpu. That will distort the time the cpu is stuck on io, since the cpu is busy with other tasks. I can reproduce it by running something like:

taskset 1 find /

in another terminal.

[krisman]

iowait is flaky and global for the cpu. Best idea I could come up is something like this to avoid spurious faulures. do you want to give that a try?

0001-test-iowait.t-Skip-if-system-is-not-quiesced-for-too.patch

[jirislaby]

0001-test-iowait.t-Skip-if-system-is-not-quiesced-for-too.patch

This still fails:

~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test> grep int.get_iowait iowait.c 
static int get_iowait(int cpu, int *user_time)
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test> ./iowait.t
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test> ./iowait.t
iowait diff too small: 1 (u=0)
test 1 failed
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test> ./iowait.t
iowait diff too small: 38 (u=0)
test 1 failed
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test> ./iowait.t
~/rpmbuild/BUILD/liburing-2.15-build/liburing-2.15/test> 

[krisman]

Perhaps consider sys time as well? Can you try this version?

Btw, since we share the same machines, where are you testing this that make it naturally reproduce?
0001-test-iowait.t-Skip-if-system-is-not-quiesced-for-too.patch