Golang runtime: behaviour change in runtime.memmove with ERMS

Go version

go version >=1.24

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='0'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/tazjin/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/tazjin/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build67497218=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/tazjin/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/tazjin/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/mdn9hnvf0qmxywbrpbywmxww2adxdn5v-go-1.24.4/share/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/tazjin/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/mdn9hnvf0qmxywbrpbywmxww2adxdn5v-go-1.24.4/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.4'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

We upgraded Go from 1.23 to 1.24, and saw a behaviour change in the pierrec/lz4 compression library which is causing some unit tests to fail on some CPUs.

git bisect on Go itself tracked this down to https://go-review.googlesource.com/c/go/+/580735 which added ERMS support to memmove on supporting CPUs.

What did you see happen?

The lz4 library's blockwise encoder is failing to decode data that could previously be decoded, failing with a generic error message. A detailed description is available in the issue we filed in the library: https://github.com/pierrec/lz4/issues/233

It's important to note that the library implements block decoding in Go assembler, and the calls to runtime.memmove around which ~something goes wrong are in there: https://github.com/pierrec/lz4/blob/v4/internal/lz4block/decode_amd64.s

We haven't tracked down the root issue yet (the reproducer in the linked issue currently still uses a lot of surrounding code, unfortunately), so it is entirely possible that some documented condition is violated in this assembler. However, I didn't find a policy on reporting potential regressions / issues of this kind, so decided to file a bug with you just in case this counts as a problem with Go itself.

What did you expect to see?

We expected no change in behaviour after upgrading to Go 1.24

Comment From: gabyhelp

Related Issues

• runtime: some kind deadlock issue with the new scheduler #4998 (closed)
• cmd/go: unable to build with a v4 module #40156 (closed)
• runtime: crash in memmove on loongson 3a4000 (mips64le) #50728
• cmd/compile: go1.21rc2: relocation R_X86_64_PC32 out of range error due to inliner changes #61218 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: randall77

I can reproduce this, but I don't know what is going on yet. Turning the memmove tests in the runtime up to 11 doesn't reveal anything.

The assembly calling memmove looks ok, I don't think there is a problem there.

Maybe the memmove assembly does a CLD (clear direction)` in more situations now, and the lz4 assembly depends on the direction flag somehow? I don't see anything obviously wrong with the lz4 assembly, but maybe worth double-checking.

Comment From: tazjin

I have stripped down the reproducer to just this: https://gist.github.com/tazjin/50d67fb95aafd1a05c2ee9453a6f48fe

    compressedBlock := []byte{
        see the gist for the full inlined data
    }

    // After removing this print statement the bug will no longer reproduce!
    fmt.Printf("first 32 bytes: %x\n", compressedBlock[:min(32, len(compressedBlock))])

    uncompressedData := make([]byte, 32768)

    // Direct call to lz4.UncompressBlock - this fails with ERMS+FSRM, and works with `GODEBUG=cpu.fsrm=off`
    resultSize, err := lz4.UncompressBlock(compressedBlock, uncompressedData)
    if err != nil {
        fmt.Printf("ERMS BUG REPRODUCED: %v\n", err)
        os.Exit(1)
    }

The inlined data in the gist is a compressed 32k block sourced from /dev/random. I observed several interesting effects, which consistently reproduce:

1. The problem is data-dependent: Not all compressed random 32k blocks trigger it, but I wasn't able to figure out a pattern.
2. Removing the fmt.Printf call will cause the problem to not reproduce anymore. I didn't investigate if other uses of that byte slice would have the same effect.
3. Moving compressedBlock to a package-level var will also cause the problem to not reproduce.

So I'd say there's definitely something strange going on!

Comment From: randall77

I can get your latest repro to fail with GODEBUG=cpu.erms=off by changing the memmove code to add MOVQ $0xaaaa, DX to the start of memmove. So it looks like something is depending on DX not being clobbered by the memmove implementation.

Comment From: randall77

Your reproducer pulls in a quite old version of lz4. It looks like it gets v2.6.1+incompatible (whatever that means) and the latest is v4.1.22. v4.1.22 doesn't have this failure. v3.3.5 does. My guess is this instruction: https://github.com/pierrec/lz4/blob/1a84cb2a9795a28b84df55a898b65b39d3d6b1f1/decode_amd64.s#L197

MOVB does not clear the upper 56 bits. You would need MOVBQZX. v4 does this store/load with MOVL which clears the upper bits.

Comment From: randall77

It appears it was more-or-less accidentally fixed by https://github.com/pierrec/lz4/commit/3135ebdb71e223316a2ff8c8b23fef8f642a690f

I don't think this is a Go bug, so closing.