What version of Go are you using (go version)?
$ go version go version go1.18.4 freebsd/amd64
Does this issue reproduce with the latest release?
Yes, but it's dependent on the FreeBSD release. i.e. issue present on FreeBSD 13.1 and greater.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/root/.cache/go-build" GOENV="/root/.config/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="-mod=vendor" GOHOSTARCH="amd64" GOHOSTOS="freebsd" GOINSECURE="" GOMODCACHE="/root/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="freebsd" GOPATH="/root/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/local/go118" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go118/pkg/tool/freebsd_amd64" GOVCS="" GOVERSION="go1.18.4" GCCGO="gccgo" GOAMD64="v1" AR="ar" CC="cc" CXX="clang++" CGO_ENABLED="1" GOMOD="/usr/local/poudriere/ports/default/net/rclone/work/github.com/rclone/rclone@v1.59.0/go.mod" GOWORK="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3160769938=/tmp/go-build -gno-record-gcc-switches" GOROOT/bin/go version: go version go1.18.4 freebsd/amd64 GOROOT/bin/go tool compile -V: compile version go1.18.4 uname -v: FreeBSD 14.0-CURRENT #3 main-n256462-79e1500276a-dirty: Thu Jun 30 19:37:24 EDT 2022 root@e6220:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG lldb --version: lldb version 14.0.5 (https://github.com/llvm/llvm-project.git revision llvmorg-14.0.5-0-gc12386ae247c) clang revision llvmorg-14.0.5-0-gc12386ae247c llvm revision llvmorg-14.0.5-0-gc12386ae247c
What did you do?
Tried to use the rclone program https://github.com/rclone/rclone to mount a remote filesystem. (rclone uses fuse behind the scenes.)
# rclone mount ~ /mnt
What did you expect to see?
(nothing) the rclone mount command should complete without error.
What did you see instead?
# rclone mount ~ /mnt 2022/07/27 17:57:57 ERROR : /mnt: Unmounted rclone mount 2022/07/27 17:57:57 Fatal error: failed to umount FUSE fs: resource temporarily unavailable
Discussion
The issue is discussed in FreeBSD bug 258056 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258056
That *BSD does not map regular files to polling is covered here https://github.com/golang/go/issues/19093
Possible patch
Patch below adds "/dev/fuse" as another pollable = false exception.
--- src/os/file_unix.go.orig 2022-07-12 11:22:57.000000000 -0400
+++ src/os/file_unix.go 2022-07-27 08:35:28.234028000 -0400
@@ -165,6 +165,10 @@
if (runtime.GOOS == "darwin" || runtime.GOOS == "ios") && typ == syscall.S_IFIFO {
pollable = false
}
+
+ if runtime.GOOS == "freebsd" && name == "/dev/fuse" { // /dev/fuse always reports ready for writing
+ pollable = false
+ }
}
}
Comment From: ianlancetaylor
Looking for the magic name /dev/fuse doesn't seem like a good approach. What if someone does ln -s /dev/fuse myfuse and then tries to open myfuse? Presumably that will fail in the same way.
Is there some way to check for a type as we already do for a FIFO?
Comment From: ianlancetaylor
CC @golang/freebsd
Comment From: dmgk
We could probably implement a function similar to devname(3), which is used by mount_fusefs(8) to check the device name. Hardcoding "kern.devname" MIB (which is [1, 2147483224] on 13.1-RELEASE) doesn't sound right so this approach would need runtime translation of the sysctl name to MIB. sysctlnametomib() is present in runtime but unexported. I'm not sure what would be the best course of action here.
Comment From: ianlancetaylor
For example, it would be fine to add sysctlnametomib to internal/syscall/unix and to call it from the os package.
Comment From: gopherbot
Change https://go.dev/cl/420235 mentions this issue: os: Disable polling for fuse devices on FreeBSD
Comment From: paulzhol
I've ended looking at this as well. I ran the bazil/fuse/examples/hellofs example, under truss, getting the same result:
```
truss -SdHf -o /tmp/truss ./hellofs -fuse.debug /tmp/tst1
2022/07/30 14:30:17 resource temporarily unavailable
The relevant part is this:
3678 100150: 0.032046504 open("/dev/fuse",O_RDWR|O_CLOEXEC,00) = 3 (0x3)
3678 100150: 0.032667484 fstat(3,{ mode=crw-rw-rw- ,inode=102,size=0,blksize=4096 }) = 0 (0x0) 3678 100150: 0.032856138 kqueue() = 4 (0x4) 3678 100150: 0.033037163 fcntl(4,F_SETFD,FD_CLOEXEC) = 0 (0x0)
3678 100150: 0.033541314 compat11.kevent(4,{ 3,EVFILT_READ,EV_ADD|EV_CLEAR,0,0,0x827253f68 3,EVFILT_WRITE,EV_ADD|EV_CLEAR,0,0,0x827253f68 },2,0x0,0,0x0) = 0 (0x0) 3678 100150: 0.033721862 fcntl(3,F_GETFL,) = 2 (0x2) 3678 100150: 0.033866169 fcntl(3,F_SETFL,O_RDWR|O_NONBLOCK) ERR#19 'Operation not supported by device'
Note how the file is registered under the netpoller, but fails to set the file into non-blocking mode. Corresponding to the following code:
https://github.com/golang/go/blob/9a2001a8cc75b1ceda17da828edcee93c9d9f42c/src/os/file_unix.go#L171-L184
We don't unregister a `pollable` file if `syscall.SetNonblock()` failed. I think that is the bug in the runtime we should be fixing.
While this is addressed (or maybe wait for a kernel fix?), this is a simple workaround to https://github.com/bazil/fuse/issues/280
```diff
diff --git a/mount_freebsd.go b/mount_freebsd.go
index 9106a18..52a1af0 100644
--- a/mount_freebsd.go
+++ b/mount_freebsd.go
@@ -56,10 +56,11 @@ func mount(dir string, conf *mountConfig) (*os.File, error) {
}
}
- f, err := os.OpenFile("/dev/fuse", os.O_RDWR, 0o000)
+ fd, err := syscall.Open("/dev/fuse", os.O_RDWR|syscall.O_CLOEXEC, 0o000)
if err != nil {
return nil, err
}
+ f := os.NewFile(uintptr(fd), "/dev/fuse")
cmd := exec.Command(
"/sbin/mount_fusefs",
By using os.NewFile() on a blocking file-descriptor, we end up with the "old" behavior.
Comment From: gopherbot
Change https://go.dev/cl/420334 mentions this issue: os: don't attempt to call syscall.SetNonblock() when kind == kindNonBlock
Comment From: paulzhol
I've sent https://go.dev/cl/420334, with it the bazil/fuse module can be made to work properly with /dev/fuse and the netpoller by going through os.NewFile + O_NONBLOCK which makes fdi go through the kindNonBlock flow.
When the kernel fuse device ends up supporting SETFL, then we can return back to regular os.OpenFile use.
diff --git a/mount_freebsd.go b/mount_freebsd.go
index 9106a18..28d4ef1 100644
--- a/mount_freebsd.go
+++ b/mount_freebsd.go
@@ -8,6 +8,8 @@ import (
"strings"
"sync"
"syscall"
+
+ "golang.org/x/sys/unix"
)
func handleMountFusefsStderr(errCh chan<- error) func(line string) (ignore bool) {
@@ -56,10 +58,11 @@ func mount(dir string, conf *mountConfig) (*os.File, error) {
}
}
- f, err := os.OpenFile("/dev/fuse", os.O_RDWR, 0o000)
+ fd, err := unix.Open("/dev/fuse", unix.O_RDWR|unix.O_NONBLOCK|unix.O_CLOEXEC, 0o000)
if err != nil {
return nil, err
}
+ f := os.NewFile(uintptr(fd), "/dev/fuse")
cmd := exec.Command(
"/sbin/mount_fusefs",
Comment From: paulzhol
Update: I believe there's a kernel issue here with the fuse device. bazil/fuse is always using blocking IO:
https://github.com/bazil/fuse/blob/fb710f7dfd05053a3bc9516dd5a7a8f84ead8aab/fuse.go#L557-L560
https://github.com/bazil/fuse/blob/fb710f7dfd05053a3bc9516dd5a7a8f84ead8aab/fuse.go#L577-L579
(Direct use of syscall.Read/Write on an os.File.Fd())
However the mere initial attempt to set the file descriptor into non-blocking mode, which reports an error:
fcntl(3,F_SETFL,O_RDWR|O_NONBLOCK) ERR#19 'Operation not supported by device'
Does switch the device into non-blocking mode, which enables it to return EAGAIN (resource temporarily unavailable) on Reads. These errors are unexpected by bazil/fuse causing the mount to fail.
If we skip this initial attempt, for example by explicitly starting from a blocking file descriptor https://github.com/golang/go/issues/54100#issuecomment-1200165500. Then the device doesn't enter non-blocking mode, and never returns EAGAIN.
Comment From: dmgk
Remaining FreeBSD fuse(4) fcntl issue is tracked here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265736
Comment From: asomers
That FreeBSD bug, which actually turned out not to be in fusefs, is now fixed. The fix will be included in the next builds of 14.0-CURRENT.
Comment From: nunotexbsd
@asomers
# restic -r restic-repo mount /mnt OK
read, copy files from /mnt OK
Thank you!