Golang runtime: feature request: Support for ASLR with the heap

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 freebsd/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/shawn/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="freebsd"
GOOS="freebsd"
GOPATH="/home/shawn/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/freebsd_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build851003260=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run a go application

What did you expect to see?

The heap mapped at a non-fixed location.

What did you see instead?

The heap mapped at a fixed location.

More information:

I've read through a previous discussion in 2012 about golang's use of deterministic memory allocations here: https://groups.google.com/forum/#!msg/golang-nuts/Jd9tlNc6jUE/qp2oyfEEfjQJ

Back then, -buildmode=pie didn't exist back then, so I figured a revisit of the topic might be warranted.

Go 1.11 introduced -buildmode=pie for FreeBSD, which HardenedBSD is able to take advantage of, given HBSD's PaX ASLR implementation. HardenedBSD is a hardened derivative of FreeBSD.

When applying -buildmode=pie with gitea, I wanted to verify that golang was able to take full advantage of HardenedBSD's robust ASLR implementation. I noticed a fixed mapping at 0xc000000000. I restarted the process, looked at the memory mappings again (procstat -v $PID) and saw 0xc000000000 again. Grepping through the golang 1.11 codebase revealed that Go allocates its heap at a fixed address, starting at 0xc000000000.

With the heap being allocated at a fixed mapping, golang applications are still not able to really utilize ASLR. Thus, -buildmode=pie is moot.

Given that the reason to implement PIE support is to take advantage of ASLR, I'm curious if perhaps updating the heap management code for ASLR support was overlooked. Or perhaps it wasn't, and this is deliberate. Or perhaps a totally different reason. Either way, could someone shed some light on this?

I submitted a post on the golang-nuts mailing list, asking the same questions. It can be found here: https://groups.google.com/d/msg/golang-nuts/pAGcmRA_V5s/2NnOPDEcBgAJ

Comment From: randall77

@aclements

Comment From: lattera

Just checking in. Any movement on this? I'm still seeing that the heap is mapped at a fixed address (0xc000000000)

Comment From: ianlancetaylor

Nothing has changed.

A randomly placed heap requires completely different code from support for Position Independent Executables. And fixed location of executable code is riskier than fixed location of non-executable data. I say this not to say that a randomly placed heap is a bad idea, but merely to observe that -buildmode=pie doesn't really have anything to do with a randomly placed heap, despite their both being based on the same underlying idea.

Comment From: lattera

I'm of the opinion that non-executable data is still executable (read: data-only attacks causing the code to take different code branches.) Fuzzers show this concept in action. Machines are weird. ;-)

In order for ASLR to be effective, everything's gotta be randomized. My original statement from years back about PIE is only tangentially related. It just doesn't make sense to support PIE without also randomizing the heap. (Not that I'm arguing to remove existing PIE support--the opposite: to include heap randomization.)

Comment From: ianlancetaylor

Since Go programs are always multi-threaded, the contents of the heap are already somewhat randomized. I can't agree that it doesn't make sense to support PIE without randomizing the heap address.

But I'm not opposed to this, it's just that somebody has to do the work. Want to volunteer?

Comment From: lattera

I'd love to, but my spare time's focused on HardenedBSD. What I can do, though, is ping the HardenedBSD community to see if anyone's interested in volunteering. I'll report back if I can find a volunteer.

Comment From: theLOICofFRANCE

Followed. I hope that one of the 65 members of @goland or @google can take care of it.

Thanks.

Comment From: walterjwhite

I'd also like to voice support for this. I'm on FreeBSD and enabled ASLR just to find that my go programs don't run. Fortunately though, I can temporarily disable ASLR without rebooting:

sysctl kern.elf64.aslr.enable=0 sysctl kern.elf32.aslr.enable=0

Comment From: lattera

I'd also like to voice support for this. I'm on FreeBSD and enabled ASLR just to find that my go programs don't run. Fortunately though, I can temporarily disable ASLR without rebooting:

sysctl kern.elf64.aslr.enable=0 sysctl kern.elf32.aslr.enable=0

That would likely be a bug in FreeBSD's AS{L}R implementation. golang applications work fine with the PaX-inspired ASLR implementation in HardenedBSD.

Comment From: emaste

That would likely be a bug ...

I have no problem running go programs on FreeBSD with ASLR enabled. We'd need some more details from @walterjwhite, but should move this discussion / bug report somewhere other than this feature request issue.

Comment From: walterjwhite

I suppose you're right.

I have some applications working with no issues in go, but I have another one which uses chromedpexecutor that drives my browser that doesn't work:

fatal error: too many address space collisions for -race mode

runtime stack: runtime.throw({0xcdaec8, 0x30}) /usr/local/go/src/runtime/panic.go:1198 +0x74 fp=0x7fffff27ffa8 sp=0x7fffff27ff78 pc=0x47d134 runtime.(mheap).sysAlloc(0x1172e80, 0x400000) /usr/local/go/src/runtime/malloc.go:686 +0x6c7 fp=0x7fffff280040 sp=0x7fffff27ffa8 pc=0x452967 runtime.(mheap).grow(0x1172e80, 0x1) /usr/local/go/src/runtime/mheap.go:1347 +0x85 fp=0x7fffff2800b0 sp=0x7fffff280040 pc=0x46eec5 runtime.(mheap).allocSpan(0x1172e80, 0x1, 0x0, 0x2e) /usr/local/go/src/runtime/mheap.go:1179 +0x1bd fp=0x7fffff280120 sp=0x7fffff2800b0 pc=0x46e99d runtime.(mheap).alloc.func1() /usr/local/go/src/runtime/mheap.go:913 +0x8c fp=0x7fffff280170 sp=0x7fffff280120 pc=0x46e3ec runtime.(mheap).alloc(0x1172e80, 0x1, 0x2e, 0x1) /usr/local/go/src/runtime/mheap.go:907 +0x8e fp=0x7fffff2801c0 sp=0x7fffff280170 pc=0x46e2ce runtime.(mcentral).grow(0x1185348) /usr/local/go/src/runtime/mcentral.go:241 +0x79 fp=0x7fffff280210 sp=0x7fffff2801c0 pc=0x45eaf9 runtime.(mcentral).cacheSpan(0x1185348) /usr/local/go/src/runtime/mcentral.go:161 +0x7e5 fp=0x7fffff280290 sp=0x7fffff280210 pc=0x45e905 runtime.(mcache).refill(0x80afa8108, 0x2e) /usr/local/go/src/runtime/mcache.go:162 +0xaa fp=0x7fffff2802d8 sp=0x7fffff280290 pc=0x45d92a runtime.(*mcache).nextFree(0x80afa8108, 0x2e) /usr/local/go/src/runtime/malloc.go:880 +0x8d fp=0x7fffff280318 sp=0x7fffff2802d8 pc=0x452a0d runtime.mallocgc(0x188, 0xcbb3e0, 0x1) /usr/local/go/src/runtime/malloc.go:1071 +0x530 fp=0x7fffff2803a8 sp=0x7fffff280318 pc=0x4530f0 runtime.newobject(0xcbb3e0) /usr/local/go/src/runtime/malloc.go:1228 +0x38 fp=0x7fffff2803d8 sp=0x7fffff2803a8 pc=0x453718 runtime.malg(0x8000) /usr/local/go/src/runtime/proc.go:4220 +0x31 fp=0x7fffff280418 sp=0x7fffff2803d8 pc=0x489131 runtime.mpreinit(0x115a0e0) /usr/local/go/src/runtime/os_freebsd.go:295 +0x29 fp=0x7fffff280438 sp=0x7fffff280418 pc=0x4795c9 runtime.mcommoninit(0x115a0e0, 0xffffffffffffffff) /usr/local/go/src/runtime/proc.go:803 +0x13a fp=0x7fffff280480 sp=0x7fffff280438 pc=0x4814ba runtime.schedinit() /usr/local/go/src/runtime/proc.go:691 +0xcf fp=0x7fffff2804e0 sp=0x7fffff280480 pc=0x480fef runtime.rt0_go() /usr/local/go/src/runtime/asm_amd64.s:212 +0x125 fp=0x7fffff2804e8 sp=0x7fffff2804e0 pc=0x4b0a65

Comment From: aclements

@walterjwhite , mind filing a new issue? It looks like you have some compatibility issue between the Go runtime and ASLR, while this issue is about enabling randomization of the Go heap, which is not about correctness.

Comment From: walterjwhite

https://github.com/golang/go/issues/51523

Comment From: gopherbot

Change https://go.dev/cl/668635 mentions this issue: runtime: add randomization to arena allocation hints

Comment From: rolandshoemaker

For mixed Go/C binaries, heap base address randomization seems like a reasonable feature, especially when ASLR is already being used to prevent abuse of the C heap.

It seems like the general consensus is, at least for 64 bit platforms, that 32 bits of entropy is a baseline.

We use anywhere between 40 and 64 bit heap addresses, depending on the architecture. When we allocate heap arenas we align them to 4MB or 64MB chunks (22 or 26 bits of alignment), depending on the platform, and when we allocate pages within these arenas we align them to 8KB chunks (13 bits of alignment).

If we consider the "base" heap address to be the address of the first byte of the first page in the first heap arena, the alignment needs restrict our ability to incorporate entropy, since we need at least 14 "fixed" bits. On (most) 64 bit platforms, this would leave us with 34 bits that can be used for randomization.

On runtime initialization we currently generate a list of heap arena hints that start at a fixed address. If we were to randomize the base heap address, we could simply randomize the first arena hint.

There would be a couple of complexities to this. First, we'd need to make it possible to call randinit before mallocinit in order to generate useful random values. Currently randinit needs to do some allocations, but this could plausibly be fixed. Secondly, we'd need to randomize the base address while still maintaining alignment of both the arena, and pages.

It seems alignment could be maintained relatively simply. One approach would be as follows:

• We take a random uint64 (e.g. from bootstrapRand, after randinit has been called), and truncate it to the number of heap address bits available (e.g. 48). This is our initial base address.
• We align the base address to the heap arena size (e.g. 64MB).
• We set the base of the first arena to this address and map it.
• We set the bottom zero bits (present from alignment, e.g. 25 bits) to another random value (i.e. by taking another random uint64, truncating it to 25 bits, right shifting it 48-25 bits, and OR'ing it with the base address), and align it to the page size (e.g. 8MB). This can also be done by just setting the bits between the heap arena alignment bit and the page alignment bit to random bits, e.g. bits 25-38 (with slightly more complex bit math).
• We set mheap_.curArena.base to the randomized page aligned base address and mark all the memory between the heap aligned address and the page aligned address as allocated in mheap_.pages. This address space will be marked PROT_NONE, meaning we waste some amount of address space, but not the actual memory itself.

Our final "base address" (the first byte of the first page of the first heap arena) then becomes our randomized page aligned address.

This approach gives us heap address bits - 14 bits of entropy. On most 64 bit platforms that gives us 34 bits of entropy, and on most 32 bit platforms this gives us 18 bits of entropy.

cc @9muir @mknyszek

Comment From: gopherbot

Change https://go.dev/cl/674835 mentions this issue: runtime: randomize heap base address

Comment From: anacrolix

I tried https://go.dev/cl/674835 to address fatal error: too many address space collisions for -race mode but no dice. I also made sure to use llvm from homebrew and avoid any forced PIE stuff I could find.

> file myexe
myexe: Mach-O 64-bit executable arm64

I don't know if there's any other way I can check why it wouldn't work.

Comment From: rolandshoemaker

In CL674835 I made it so you cannot enable heap base randomization when using race mode, since the TSAN runtime requires the heap be in a very specific region ([0x00c000000000, 0x00e000000000)), and because I'm not hugely convinced that using these two things together makes a ton of sense.

I suspect the too many address space collisions issue is orthogonal to the general purpose of heap base randomization, likely the ASLR of the C portion of the program is just stamping over the address space the TSAN runtime is expecting to use? I'm not sure there is anything we can do about this on our end.