Proposal Details

Passing syscall.O_NONBLOCK (only defined on Unix OSes) to os.Open opens the file descriptor in nonblocking mode. This flag is mostly useful when working with named pipes, as they are normally used for asynchronous inter-process communication. There are 680 hits of syscall.O_NONBLOCK on GitHub (prompt).

It is currently not possible to use the os package to open a named pipe for non-blocking I/O (aka overlapped I/O) on Windows, as the syscall package doesn't define O_NONBLOCK on that port.

The os package will start supporting overlapped I/O starting in Go 1.25 (see CL 661795 and #19098), yet there will be no way to open a file for overlapped I/O without using the x/sys/windows (or syscall) packages. Note that, on Windows, it is not possible to change a file descriptor to be non-blocking once it has been created, that property must be specified up-front.

I propose to promote syscall.O_NONBLOCK to the os package and map it to FILE_FLAG_OVERLAPPED on Windows. We could also just define syscall.O_NONBLOCK for Windows, but the syscall API is frozen and IMO that flag makes sense to be in the os package, as it will now be portable across most of the supported OSes (except on plan9 and wasi?).

--

Edit: Updated proposal:

package syscall

const O_OVERLAPPED = 0x04000 // the file or device is being opened or created for asynchronous I/O.

This const will also be added to golang.org/x/sys/windows.

See https://github.com/golang/go/issues/73676#issuecomment-2918713564.

@golang/windows @golang/proposal-review

Comment From: adonovan

Would it be possible to achieve the same goal by implementing the existing syscall.SetNonblock(fd Handle, blocking bool) function, which is currently a no-op? That wouldn't require a proposal.

You might want to implement this proposal nonetheless, but SetNonblock may be a way to make immediate progress.

Comment From: qmuntal

Would it be possible to achieve the same goal by implementing the existing syscall.SetNonblock(fd Handle, blocking bool) function, which is currently a no-op? That wouldn't require a proposal.

Unfortunately that function can't be implemented on Windows. One can only set the blocking mode when creating the file descriptor (aka handle). There is no reliable way to change it later on.

Comment From: qmuntal

@neild this new flag would also affect the os.Root.OpenFile flag, as it accepts the same paramters as os.CreateFile.

Comment From: neild

What would the user-visible effect of setting O_NONBLOCK be? Are there subtly different behaviors on Unix and Windows systems? How would we document the flag?

Comment From: qmuntal

What would the user-visible effect of setting O_NONBLOCK be?

On Windows, the main effect is that I/O will not block and will be cancelable, either for named pipes and normal files. On Linux, that will make os.OpenFile (and friends) not to wait for the other side of the named pipe to connect and will also make named pipes use non-blocking cancelable I/O without having to call syscall.SetNonblock, although this last thing is already happening when using the os package.

Are there subtly different behaviors on Unix and Windows systems?

I can think on several differences:

  • On Unix, one can change the blocking mode after the file descriptor has been created, while that's not possible on Windows. This makes passing O_NONBLOCK to os.OpenFile somehow a more relevant action on Windows, as that can't be undone. Some code down the line might not be prepared to handle overlapped I/O (for example because it is using direct syscalls), so it won't be able to read or write the file.
  • On Unix, when opening a named pipe with O_NONBLOCK, the open syscall doesn't block waiting for the other end to connect. On Windows, that's the behavior you always get with CreateFileW, you can't instruct it to wait for the other end of the pipe (you have to use ConnectNamedPipe to achieve that). So using O_NONBLOCK would make the code more portable.
  • The types of files that support nonblocking I/O might be different depending on the OS. For example, Windows supports overlapped I/O for disk files, pipes and sockets, but not for consoles. On Linux, nonblocking I/O is mostly targeted for pipes and special devices.

How would we document the flag?

I don't think the documentation of the flag should go into that much detail, as this features are well documented online. Here is my proposal (syscall.O_NONBLOCK should be replaced with an unexported value on Windows):

package os

const O_NONBLOCK = syscall.O_NONBLOCK // open for nonblocking I/O in Unix and for overlapped I/O on Windows.

Comment From: apparentlymart

Is it currently true that including O_NONBLOCK on Unix can cause operations to fail with the error syscall.EAGAIN/syscall.EWOULDBLOCK if the operation cannot complete immediately? (That's the behavior I'd expect from the underlying system calls, but I've not personally tried this when working through the os package APIs.)

If so, would you expect similar situations on Windows to emulate that Unix-like behavior -- returning exactly the same error code -- or would someone writing portable code need to write defensive error handling to properly handle this situation?

Is it justified to add a higher-level error value (similar to fs.ErrNotExist) that is guaranteed to be usable with errors.Is to recognize the "would block" situation across all platforms, without having to import anything from syscall?

Comment From: neild

I'm afraid I still don't really understand when you want to set O_NONBLOCK in a Go program. (I know it's useful, I just don't understand when.)

Are there programs that run on Unix and Windows that will want to set O_NONBLOCK on both?

A documentation concern is that users may reasonably expect that setting O_NONBLOCK on a file will cause reads and writes to return immediately if they can make no progress. I don't believe that's how non-blocking files work in the runtime, however; the underlying FD may be non-blocking, but Read and Write are still blocking operations. (Am I correct about that?)

Comment From: adonovan

A documentation concern is that users may reasonably expect that setting O_NONBLOCK on a file will cause reads and writes to return immediately if they can make no progress. I don't believe that's how non-blocking files work in the runtime, however; the underlying FD may be non-blocking, but Read and Write are still blocking operations. (Am I correct about that?)

In effect, yes: Open on UNIX always creates non-blocking files, even without NONBLOCK, since the runtime poller requires it. (See newFile in os/file_unix.go, which calls SetNonblock if not already done at open(2).)

Comment From: neild

My main concerns are:

  1. If we have os.O_NONBLOCK, it shouldn't have completely different effects on different platforms. If this is a case where Unix programs might want to set O_NONBLOCK under some circumstances, and Windows programs might want to set it in entirely different circumstances, then that seems confusing.
  2. "O_NONBLOCK" is a very confusing name if it doesn't actually make a file non-blocking. "The underlying file descriptor is nonblocking, but all file operations are still blocking" is not obvious at all. This isn't that bad when it's stuffed off in syscall, but the os package aspires to offer a bit more abstraction.

I'm not necessarily opposed to os.O_NONBLOCK if it has a suitable cross-platform meaning. We'll just need to be careful about the documentation.

syscall.O_OVERLAPPED might be an alternative, if the meanings are substantially different between Unix and Windows.

Comment From: qmuntal

Is it currently true that including O_NONBLOCK on Unix can cause operations to fail with the error syscall.EAGAIN/syscall.EWOULDBLOCK if the operation cannot complete immediately?

@apparentlymart Yes, but the os.File implementation deals with those errors by waiting until the operation is ready to be consumed, so that is transparent to the user.

syscall.O_OVERLAPPED might be an alternative, if the meanings are substantially different between Unix and Windows.

Your gut feeling @neild may be right: we don't understand how os.O_NONBLOCK would be used as a portable flag, even more because until Go 1.25 os.File won't support overlapped I/O. I'm OK postponing os.O_NONBLOCK until we have more data if there is an easy way to create an overlapped os.File on Windows. syscall.O_OVERLAPPED would work.

I'm afraid I still don't really understand when you want to set O_NONBLOCK in a Go program.

O_NONBLOCK is mostly useful when working with named pipes. On Unix, you would use it in case you don't want the open syscall to block a system thread waiting for the other end of the pipe to connect. On Windows, in case you don't want I/O operations to block a system thread while running.

If we have os.O_NONBLOCK, it shouldn't have completely different effects on different platforms. If this is a case where Unix programs might want to set O_NONBLOCK under some circumstances, and Windows programs might want to set it in entirely different circumstances, then that seems confusing.

I get your point now. Setting O_NONBLOCK makes in fact the code more portable, as it aligns the Windows and Unix behaviors: both will do I/O in a nonblocking (which is not the case on Windows without the flag) manner and non will wait until the other end of the pipe is connected (which is not the case on Unix without the flag).

On the other hand, a Unix developer might want to set O_NONBLOCK just to make the open call non-blocking, but still want the file descriptor to act in a blocking manner, therefore call syscall.SetNonblock to make it blocking. That's not possible on Windows, so it would be a limitation on what one can do with os.O_NONBLOCK.

"O_NONBLOCK" is a very confusing name if it doesn't actually make a file non-blocking.

I don't see how that could happen.

Comment From: aclements

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — aclements for the proposal review group

Comment From: neild

"O_NONBLOCK" is a very confusing name if it doesn't actually make a file non-blocking.

I don't see how that could happen.

My intuitive understanding of "nonblocking file" is that reading from the file when there is no data available returns immediately. So if f is a non-blocking pipe, n, err := f.Read(buf) will return n == 0 && err != nil when no data is available. That's not what happens, though, because os.File operations are always blocking, regardless of whether the underlying file descriptor is or not.

So I don't think we should have a symbol named os.O_NONBLOCK, because a reasonable interpretation of what it does is wrong.

Reading through this issue again, I think there are a couple things that this proposal is trying to address.

First: On Windows there should be a way to open a file that uses overlapped I/O. Adding a symbol to the syscall package seems like a reasonable way to achieve this (maybe syscall.O_OVERLAPPED). One question here: Is there a way to do this automatically? You mentioned that you might do this to avoid I/O operations blocking a system thread while running. On Unix, we always create non-blocking files. Should we always use overlapped I/O on Windows (possible just for named pipes)?

Second (I think): There should be a portable way to open a named pipe that doesn't block until the other side of the pipe has connected. Maybe there should be a new os package API for this. I don't think we should call it os.O_NONBLOCK, because as described above that seems confusing, but perhaps some other name? os.O_OPEN_NAMED_PIPE_IMMEDIATELY? os.O_IMMEDIATE?

Comment From: qmuntal

My intuitive understanding of "nonblocking file" is that reading from the file when there is no data available returns immediately. So if f is a non-blocking pipe, n, err := f.Read(buf) will return n == 0 && err != nil when no data is available. That's not what happens, though, because os.File operations are always blocking, regardless of whether the underlying file descriptor is or not.

Oh, I get your point now. I see how os.O_NONBLOCK is indeed counterintuitive!

On Windows there should be a way to open a file that uses overlapped I/O. Adding a symbol to the syscall package seems like a reasonable way to achieve this (maybe syscall.O_OVERLAPPED)

That makes sense. I'll change the proposal title and content to propose syscall.O_OVERLAPPED instead.

One question here: Is there a way to do this automatically?

This was the goal of #19098. The problem is that we can't downgrade the file handle to non-overlapped when returning it from File.Fd, and callers might not be prepared to handle overlapped handles. See this comment: https://github.com/golang/go/issues/19098#issuecomment-300878859. On the other hand, we now have the GODEBUG mechanism to opt-out from breaking changes, so we could revisit this approach. Anyway, syscall.O_OVERLAPPED was value on its own, so let's leave using it by default on named pipes for another proposal.

Second (I think): There should be a portable way to open a named pipe that doesn't block until the other side of the pipe has connected.

This was more like a nice side effect of os.O_NONBLOCK than something I really want to have in os. Writing portable code when working with named pipes is hard given the different API semantics provided on Unix and Windows. Let's leave this for a future proposal that tackles named pipes holistically (if that's possible).

Comment From: aclements

Given that the updated proposal is to add a flag to syscall, how does syscall.O_OVERLAPPED relate to syscall.FILE_FLAG_OVERLAPPED?

Comment From: neild

syscall.O_OVERLAPPED is a flag passed to os.OpenFile, and syscall.FILE_FLAG_OVERLAPPED is a flag passed to syscall.CreateFile (which is a direct wrapper of CreateFileA).

Or to put it another way, you would pass syscall.O_OVERLAPPED to os.OpenFile to make it pass FILE_FLAG_OVERLAPPED to the underlying syscall.

An almost-certainly-terrible idea which just occurred to me is that perhaps we could make these part of the same bit set.

All the FILE_FLAG_* flags appear to use the high 12 bits of a uint64: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea

The made-up os.O_* flag values for Windows use the low 16 bits: https://go.googlesource.com/go/+/refs/tags/go1.24.4/src/syscall/types_windows.go#35

So we could just support os.OpenFile(name, syscall.FLAG_FILE_OVERLAPPED, mode). We could even pass through other reasonable Windows flags such as FILE_FLAG_DELETE_ON_CLOSE.

This would become exciting if Windows defines any flags conflicting with the invented O_ values, although we could possibly work around the problem so long as we never need to support more than 64 flags in total.

Comment From: qmuntal

An almost-certainly-terrible idea which just occurred to me is that perhaps we could make these part of the same bit set.

I like this terrible idea 😄

All the FILE_FLAG_* flags appear to use the high 12 bits of a uint64:

It uses the high 16 bits of a uint32 (not uint64). It's just that there are some undocumented flags. The low 16 bits are reserved for the file attributes (which roughtly correspond to file permissions on POSIX).

This would become exciting if Windows defines any flags conflicting with the invented O_ values, although we could possibly work around the problem so long as we never need to support more than 64 flags in total.

Fun fact: The CreateFile API can't accept flags other than the ones already defined, as the dwFlagsAndAttributes can't hold more values. This is why CreateFile2 was defined, it separates the attributes from the flags so both can hold more values.

Comment From: neild

So the revised idea would be:

On Windows, os.OpenFile passes through bits 16-31 of its flags parameter (the high bits of a uint32) to CreateFile. We avoid putting any synthetic O_ constants in those bits. This allows passing FILE_FLAG_* values directly to OpenFile.

The syscall package currently defines the following O_ constants that conflict with those bits:

        O_CLOEXEC      = 0x80000
        o_DIRECTORY    = 0x100000   // used by internal/syscall/windows
        o_NOFOLLOW_ANY = 0x20000000 // used by internal/syscall/windows
        o_OPEN_REPARSE = 0x40000000 // used by internal/syscall/windows
        o_WRITE_ATTRS  = 0x80000000 // used by internal/syscall/windows

The lowercase o_ ones are internal, and I think we can move those into bits 33-63. (OpenFile takes an int parameter, but I think the internal o_ constants are only passed to internal functions that we can specify as taking a uint64.)

That leaves O_CLOEXEC, which I think we might need to redefine. (I don't see a FILE_FLAG_* value using 0x80000 in the OpenFile documentation, but from what you say those bits are entirely in use.) There seem to be some unused bits in the low 16, so that's feasible. That won't break anyone who is using the symbol, but it's technically a backwards-incompatible change.

Comment From: apparentlymart

Given that the os.File abstraction makes certain assumptions about how the underlying filehandle is configured in order to cooperate with the runtime, is actually safe to blindly pass through flags to the underlying API?

That is, could any currently-defined flag cause the underlying handle to behave in a way that breaks the abstraction in a non-obvious way?

I'd personally feel inclined to constrain this only to flags that are known to be safe, with any others returning an error until enabled by a future proposal, but that is primarily me just being conservative because I'm not a Windows API expert; if those who know better feel certain that it's safe then of course I concede this point.