I am experiencing frequent test TIMEOUTs in version lines 7.2, 7.4 and 8.0, when I run them in Gentoo test environment. I noticed that after the latest security release of versions 7.2.10, 7.4.5 and 8.0.3. I don't think that I experienced the similar timeouts when I was testing the previous security batch, when 7.2.9, 7.4.4 and 8.0.2 was released but they are now also affected. grep -C5 TIMEOUT redis-{7.2.10,7.4.5,8.0.3}/temp/build.log
gives me this:
redis-7.2.10/temp/build.log-[ignore]: hash with one huge field: large memory flag not provided
redis-7.2.10/temp/build.log-[91/91 done]: violations (1 seconds)
redis-7.2.10/temp/build.log-Testing solo test
redis-7.2.10/temp/build.log-[ok]: Active defrag (46825 ms)
redis-7.2.10/temp/build.log-[skip]: Active defrag eval scripts
redis-7.2.10/temp/build.log:[TIMEOUT]: clients state report follows.
redis-7.2.10/temp/build.log-sock561b0c21d610 => (IN PROGRESS) Active defrag big keys
redis-7.2.10/temp/build.log-Killing still running Redis server 1751
redis-7.2.10/temp/build.log-Killing still running Redis server 1797
redis-7.2.10/temp/build.log-Killing still running Redis server 1794
redis-7.2.10/temp/build.log-Killing still running Redis server 1967
--
redis-7.2.10/temp/build.log- 0 seconds - bitops-large-memory
redis-7.2.10/temp/build.log- 1 seconds - violations
redis-7.2.10/temp/build.log-
redis-7.2.10/temp/build.log-!!! WARNING The following tests failed:
redis-7.2.10/temp/build.log-
redis-7.2.10/temp/build.log:*** [TIMEOUT]: clients state report follows.
redis-7.2.10/temp/build.log-Cleanup: may take some time... OK
redis-7.2.10/temp/build.log- * ERROR: dev-db/redis-7.2.10::gentoo failed (test phase):
redis-7.2.10/temp/build.log- * Failed to run command: ./runtest
redis-7.2.10/temp/build.log- *
redis-7.2.10/temp/build.log- * Call stack:
--
redis-7.4.5/temp/build.log-[ok]: Active defrag big keys: cluster (28088 ms)
redis-7.4.5/temp/build.log-[ok]: Active defrag pubsub: cluster (35031 ms)
redis-7.4.5/temp/build.log-[ok]: Active Defrag HFE: cluster (8165 ms)
redis-7.4.5/temp/build.log-[ok]: Active defrag main dictionary: standalone (48346 ms)
redis-7.4.5/temp/build.log-[ok]: Active defrag eval scripts: standalone (5361 ms)
redis-7.4.5/temp/build.log:[TIMEOUT]: clients state report follows.
redis-7.4.5/temp/build.log-sock5555f20636e0 => (IN PROGRESS) Active defrag big keys: standalone
redis-7.4.5/temp/build.log-Killing still running Redis server 20333
redis-7.4.5/temp/build.log-
redis-7.4.5/temp/build.log- The End
redis-7.4.5/temp/build.log-
--
redis-7.4.5/temp/build.log- 162 seconds - integration/replication-psync
redis-7.4.5/temp/build.log- 1 seconds - bitops-large-memory
redis-7.4.5/temp/build.log-
redis-7.4.5/temp/build.log-!!! WARNING The following tests failed:
redis-7.4.5/temp/build.log-
redis-7.4.5/temp/build.log:*** [TIMEOUT]: clients state report follows.
redis-7.4.5/temp/build.log-Cleanup: may take some time... OK
redis-7.4.5/temp/build.log- * ERROR: dev-db/redis-7.4.5::gentoo failed (test phase):
redis-7.4.5/temp/build.log- * Failed to run command: ./runtest
redis-7.4.5/temp/build.log- *
redis-7.4.5/temp/build.log- * Call stack:
--
redis-8.0.3/temp/build.log-[ignore]: SETBIT values larger than UINT32_MAX and lzf_compress/lzf_decompress correctly: large memory flag not provided
redis-8.0.3/temp/build.log-[95/95 done]: bitops-large-memory (0 seconds)
redis-8.0.3/temp/build.log-Testing solo test
redis-8.0.3/temp/build.log-[ok]: Active defrag main dictionary: cluster (53920 ms)
redis-8.0.3/temp/build.log-[ok]: Active defrag eval scripts: cluster (7182 ms)
redis-8.0.3/temp/build.log:[TIMEOUT]: clients state report follows.
redis-8.0.3/temp/build.log-sock55bd580abee0 => (IN PROGRESS) Active defrag big keys: cluster
redis-8.0.3/temp/build.log-Killing still running Redis server 23878
redis-8.0.3/temp/build.log-
redis-8.0.3/temp/build.log- The End
redis-8.0.3/temp/build.log-
--
redis-8.0.3/temp/build.log- 240 seconds - integration/replication-psync
redis-8.0.3/temp/build.log- 0 seconds - bitops-large-memory
redis-8.0.3/temp/build.log-
redis-8.0.3/temp/build.log-!!! WARNING The following tests failed:
redis-8.0.3/temp/build.log-
redis-8.0.3/temp/build.log:*** [TIMEOUT]: clients state report follows.
redis-8.0.3/temp/build.log-Cleanup: may take some time... OK
redis-8.0.3/temp/build.log- * ERROR: dev-db/redis-8.0.3::gentoo failed (test phase):
redis-8.0.3/temp/build.log- * Failed to run command: ./runtest
redis-8.0.3/temp/build.log- *
redis-8.0.3/temp/build.log- * Call stack:
Interestingly, 6.2.19 versions is not affected, therefore, I bisected the code between 8.0.3 and 6.2.19. The git bisect pointed to the commit 98b3f52599cc - add test suite infra to test RESP3 attributes (#10247)
and tests pass if I revert this change in 8.0.3. The tests are executed with following parameters in gentoo ebuild:
./runtest --clients 16 --skiptest '/Active defrag for argv retained by the main thread from IO thread.*' --skipunit unit/oom-score-adj --skiptest 'CONFIG SET rollback on apply error' --tls
and it is related to --tls
parameter, because tests pass without it.
Comment From: sundb
@arkamar this timeout has also frequently happended in github actions recently. Could you share your detailed steps to reproduce it?
Gentoo test environment
Do you mean run this test on Gentoo?