Golang os: Root.MkdirAll can return "file exists" when called concurrently on the same path

Go version

go version go1.25.0 linux/amd64

Output of `go env` in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v3'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/database64128/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/database64128/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2391249113=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/database64128/repos/modpack-dl-go/go.mod'
GOMODCACHE='/home/database64128/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/database64128/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/database64128/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I have a file downloading program that fetches a file list and spins up 32 worker goroutines to download all listed files. Before downloading, it first tries to open or create the file at the destination to see if it can be skipped:

// createFile creates the file at the given path.
// The parent directory will be created if it doesn't exist.
// It returns the opened created file or an error.
func createFile(path string) (*os.File, error) {
    f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
    if err != nil {
        if err = os.MkdirAll(filepath.Dir(path), 0755); err != nil {
            return nil, err
        }
        return os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
    }
    return f, nil
}

After I upgraded to Go 1.25 and converted it to use os.Root (#67002):

// createFile creates the file at the given path.
// The parent directory will be created if it doesn't exist.
// It returns the opened created file or an error.
func createFile(root *os.Root, path string) (*os.File, error) {
    f, err := root.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
    if err != nil {
        if err = root.MkdirAll(filepath.Dir(path), 0755); err != nil {
            return nil, err
        }
        return root.OpenFile(path, os.O_RDWR|os.O_CREATE, 0644)
    }
    return f, nil
}

What did you see happen?

createFile now has a chance of returning an error file exists. This is likely caused by:

https://github.com/golang/go/blob/ffc85ee1f1c865c953920f966a8401d963b102ca/src/os/root_openat.go#L126-L136

~~The error path at L135 needs to double-check whether the directory exists, like os.MkdirAll does:~~

https://github.com/golang/go/blob/ffc85ee1f1c865c953920f966a8401d963b102ca/src/os/path.go#L54-L65

Otherwise, if the directory was created between rootOpenDir and mkdirat by another goroutine, os.Root.MkdirAll would fail.

~~I can send a CL to fix this if you are OK with this approach.~~ @neild

Update: Opened CL 698215.

What did you expect to see?

os.Root.MkdirAll does not return any error.

Comment From: gabyhelp

Related Issues

Related Code Changes

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Comment From: gopherbot

Change https://go.dev/cl/698215 mentions this issue: os: fix Root.MkdirAll to handle race of directory creation

Comment From: database64128

@gopherbot please open backport issues. The fix is just an additional error check. The workaround involves calling os.Root.MkdirAll in a loop, which doesn't look very safe:

for {
    if err := root.MkdirAll(path, 0755); err != nil {
        if errors.Is(err, os.ErrExist) {
            continue
        }
        return nil, err
    }
    break
}

You can't put a limit on the number of retries, unless you know in advance how many path components are there.