Proposal Details
Recently, there have been more and more news like https://socket.dev/blog/11-malicious-go-packages-distribute-obfuscated-remote-payloads or https://alexandear.github.io/posts/2025-02-28-malicious-go-programs/ etc I propose to implement at the api level the ability to monitor and modify a variety of system calls. Following the example of how it's already done in python https://docs.python.org/3/library/audit_events.html But it's done very poorly there, and it would be useful to implement a full-fledged hook subsystem like
audit.Handle(os.RemoveAll, func(path string) error {
fmt.Println("Path remove request for:", path)
switch path {
case "/bin":
return nil // deny
case "/tmp/old":
return os.Rename(path, "/tmp/new") // replace call
case "/home":
path = "/tmp/trash" // replace argument
}
return os.RemoveAll(path)
})
or
audit.Handle(http.Get, func(url string) (resp *http.Response, err error) {
fmt.Println("Http get request for:", url)
switch url {
case "some.malware.host":
return nil, nil
default:
url = "http://127.0.0.1:8080"
}
return http.Get(url)
})
All this can be done anyway if start inject handler code directly in the golang source code, but why spoil it if you can add a separate powerful subsystem.
Comment From: seankhliao
I believe this is generally considered out of scope. See also #50632
Comment From: mwriter
I believe
And what right do you have to put your faith above technical discussions? You put your personal preferences above the interests of the community.
See also #50632
There's a completely different question there
Comment From: zigo101
It looks seankhliao thinks he is the new Go leader. :D
Comment From: dominikh
There's a completely different question there
They address the same need and both suffer the same limitation, see https://github.com/golang/go/issues/50632#issuecomment-1013783694.
It is not the process's job to audit itself.
Comment From: mwriter
It is not the process's job to audit itself.
The process doesn't do anything by itself, the hooks are created by the programmer to monitor events. If the hooks are not added manually, then there will be no additional impact on performance. This is a completely different approach compared to the one proposed there by writing rules in go.mod by analogy with AndroidManifest.xml. By the way, Android has a similar system of hooks at runtime for a long time. Even if it's an outside party. And the idea itself fits perfectly conceptually, whether it's in a language or an operating system.
Comment From: mwriter
And most importantly, anyone can implement this subsystem by injecting their own handlers to the golang source codes. This is the main argument in favor of the easy possibility of implementing such a subsystem. It was much more difficult to implement this in python. But even there it was done. Albeit very clumsily.
Comment From: zigo101
@seankhliao
Did you hide @mwriter's comment: https://github.com/golang/go/issues/75059#issuecomment-3194478315? It is hard to say that comment is off-topic. It contains relevant explanation.
Please be polite to Go community members.
Comment From: ianlancetaylor
The proposed facility would not be sufficient to prevent malicious packages from doing nefarious things. Go provides too many mechanisms for programs to invoke system calls, such as assembler code and C code. With Go, preventing in-process behavior can only be reliably done out of process, such as by sandboxing. It can't be done in-process.
A more feasible approach for handling malicious packages (other than sandboxing) might be analyzers that verified that imported packages only used certain facilities. Those analyzers could explicitly call out assembler and C code which would have to be examined manually.
Comment From: mwriter
The proposed facility would not be sufficient to prevent malicious packages from doing nefarious things.
But it's not just about security. In python auditing doesn't cover security issues at all either. It's mainly about the possibility of general monitoring, logging and tracing. First of all, it's about what dynamic execution analysis tools golang provides. There are still a lot of advantages from such an implementation. Moreover, it's not difficult to add the simplest implementation, unlike python.
Comment From: mwriter
Those analyzers could explicitly call out assembler and C code which would have to be examined manually.
By the way, the question is in this regard. After all, during golang runtime, is there any way to detect that C or assembler codes are being called? If there are variants, then can inject hooks on generally switch to such calls and interrupt their execution according to the proposed scenario. Then in this case, even the most complete security is possible, unlike python. Because standard APIs will be processed in separate hooks, and the transition to doing anything non-standard can simply be blocked in general hook. And as a result, we will get a semblance of a virtual machine at the golang runtime level :) Or, when compiled, do all the codes turn into a single binary assembler code that cannot be separated by the original type?
Comment From: mwriter
@cugu
The only exceptions here would be the use of C and assembler code which can be detected as well.
Here, you wrote that once. Can you tell in more detail how easy it is to do this in golang? Maybe you'll be interested in this discussion as a whole. Because if it's possible to implement checks at the level of individual hooks, then it will be possible to extend to the most complex scenarios at the level of the entire module management, which you suggested.
Comment From: ianlancetaylor
But it's not just about security.
Well, OK, but the examples in the original proposal seem to all be about security.
For that matter it occurs to me that implementing this feature would tend to make Go programs less secure against malicious packages, because any package that could call audit.Handle could radically change the behavior of the program.
is there any way to detect that C or assembler codes are being called?
In the current implementation: C: yes, in general, barring some complex tricks that a package could engage in. Assembler: no.
Comment From: mwriter
Well, OK, but the examples in the original proposal seem to all be about security.
And how exactly did you understand that the examples are related to security? ;) But the essence of my examples is solely in the monitoring cases. Here the user wants to track all system golang calls from all code. How can he do this now in runtime without a full static analysis of the code (which can also be obfuscated)?
because any package that could call
audit.Handlecould radically change the behavior of the program
Yes, this is an important point.
We will have to make a restriction that only the main module can call it. It will be easy to do this check during compilation, because golang builds a complete dependency graph anyway. It will be enough to check that a package audit has been imported somewhere in the submodules.
In the current implementation: C: yes, in general, barring some complex tricks that a package could engage in
And how exactly can this be tracked in runtime?
Assembler: no
So we'll omit the assembler inserts. Its can be tracked through static code analysis. And in any case, the main thing isn't security but monitoring. Or if we can't provide complete security or monitoring, then let's not provide anything at all? ;) I understand that even Google doesn't have endless resources. In python, it would be exactly the same if two people hadn't implemented it at will. And everything is done very clumsily there, considering that python has much more dynamic features. But I think if this is implemented, it will be a functionality that distinguishes Golang from other languages along with goroutines and channels.
Comment From: ianlancetaylor
Here the user wants to track all system golang calls from all code. How can he do this now in runtime without a full static analysis of the code (which can also be obfuscated)?
Run the program under strace and examine the output. Or for more control run the program in a sandbox and verify the system calls.
Or if we can't provide complete security or monitoring, then let's not provide anything at all? ;)
Yes. Providing an incomplete facility leads to confusion and misunderstanding, especially for a feature that appears at first glance to be related to security.
Comment From: mwriter
Run the program under strace and examine the output
For example, in Android or Windows? ;)
Yes. Providing an incomplete facility leads to confusion and misunderstanding, especially for a feature that appears at first glance to be related to security.
It would be interesting to reopen the issue and collect likes and dislikes in a month or two. And we would have been convinced ;) Having some functionality and choice is always better than not having them :)
Comment From: dominikh
And how exactly did you understand that the examples are related to security? ;)
It is implied by the term "audit". The purpose of an audit is to verify the absence of wrong-doing. Furthermore, you yourself say
Here the user wants to track all system golang calls from all code. How can he do this now in runtime without a full static analysis of the code (which can also be obfuscated)?
Why is obfuscation a concern if this isn't for security purposes? It seems to me that you want to reliably "monitor", i.e. audit, which syscalls get used how. That is security-relevant, and exactly what the proposed feature cannot do. Actual auditing of syscalls is impossible to do reliably in a single address space in a language like Go.
For example, in Android or Windows? ;)
strace or procmon. There are existing and better solutions on all operating systems that Go supports.
Having some functionality and choice is always better than not having them :)
That does not match Go's ethos. All functionality has a cost, in particular for the maintainers who will have to maintain it indefinitely. It also adds cost for users, especially when the feature is ripe with drawbacks and tradeoffs and won't work for its intended purpose.
Comment From: mwriter
It is implied by the term "audit". The purpose of an audit is to verify the absence of wrong-doing. Furthermore, you yourself say
Names are the last thing to get attached to. Python has them, but they themselves try in every possible way to avoid them and name the main purpose monitoring. In general, I need functionality and let Ian come up with the names :)
Actual auditing of syscalls is impossible to do reliably in a single address space in a language like Go.
But it is definitely possible to monitor all calls from Go API.
There are existing and better solutions on all operating systems that Go supports
Are you sure you're looking at everyone right now? There are also about two dozen supported systems. And are you sure that this monitoring will be so easy? But even supposing if so. But how will this help me in monitoring specific Go API calls?
All functionality has a cost
I totally agree with that. But I categorically disagree that the issues, and the issues are completely nontrivial and require comprehensive discussion, are closed within a few minutes. Sean Liao isn't Rob Pike, or even Ian Taylor. Can someone at least tell me what he did for Golang? But he behaves like he's the main one here, a well-deserved and recognized specialist. Has he ever borne any responsibility for closing a non-trivial issue that needs to be discussed?
Comment From: randall77
But he behaves like he's the main one here, a well-deserved and recognized specialist. Has he ever borne any responsibility for closing a non-trivial issue that needs to be discussed?
Sean does a lot of great work gardening new issues. Finding and closing duplicates, etc. His word is not final. No one's word is final. Closing can be reversed if needed.
That said, I don't think this issue needs to be reopened (or I would have). Similarly for others who commented on this issue.
There's a lot of places to discuss your ideas about Go: https://go.dev/wiki/Questions If something like a consensus that it is useful and implementable comes out of such a discussion, we can reopen (or start a new issue).
We currently have 9,179 open issues. For "open" to mean anything we need to get that number smaller, not let it keep getting larger. For proposals "open" means ~"has a chance to be approved by the proposal committee". It is a judgement call, but we (Sean and others) need to make those judgement calls. Pretty much every proposer doesn't like to have their proposal closed, but we have to be able to abandon unlikely proposals quickly if we ever want to keep up with the proposal volume.
Comment From: mwriter
Finding and closing duplicates, etc.
This is a routine side job. And I'm talking about creative, highly intellectual work, which has made Golang a world-class language. I'm sure his role in this is extremely small. And I personally have big questions about his competence even for this job.
That said, I don't think this issue needs to be reopened (or I would have). Similarly for others who commented on this issue.
I absolutely don't insist. But it would still be interesting to keep this issue open for a month or two, and collect likes and dislikes. But if Ian says that there are no resources for this anyway, then there is no :)
There's a lot of places to discuss your ideas about Go
Do you really want to discuss this here again? I'll say it simply and directly: this isn't the case, and I can consistently prove it.
We currently have 9,179 open issues
For some reason, I have confidence that I'm raising completely nontrivial issues, and according to some criteria its may well be among the open ones, at least for more than five minutes ;)
but we (Sean and others) need to make those judgement calls
But for some reason, he is constantly in the role of such an "evil policeman" :) This is definitely not an accident, which means either he's essence, or a purposefully assigned role ;) In any case, it's much more useful for me to hear your negative verdict with minimal argument than his words "I believe" :) And I won't hide it, most of all I like Ian's detailed argumentation on questions of any complexity.
Comment From: zigo101
@seankhliao indeed did much work which other people have not interests to do. Mostly good job. But it looks he also often closed good issues, which he didn't/doesn't understand well or at all. Just two of them:
- https://github.com/golang/go/issues/70813
- https://github.com/golang/go/issues/50167 then https://github.com/golang/go/issues/61725.
Comment From: mwriter
@zigo101 As a result, the discussion turned to a discussion of his person. He has a lot of undeserved fame :( I would prefer to discuss the actual issue of golang call monitoring. But it seems there are few interested parties, and Google has failed at this stage of trying to create a secure golang environment, and they aren't going to return to it. And Keith also clearly clarified the interests of the team in minimizing open issues, and in general everything is clear.
Comment From: mwriter
@ianlancetaylor @randall77 @dominikh @zigo101
Simple implementation :)
- Create the folder
go/src/auditand create the filego/src/audit/audit.go
package audit
import (
"iter"
"slices"
"sync"
)
const (
Invalid = "Invalid audit hook"
Hook = "audit.Handle"
RemoveAll = "os.RemoveAll"
)
var (
hooks *sync.Map
)
func Enabled() bool {
return hooks != nil
}
func Get(name string) (hook any) {
if h, ok := Load(name); ok {
hook = h
}
return hook
}
func handle(name string, hook any) {
hooks.Store(name, hook)
}
func Handle(name string, hook any) {
if !Enabled() {
hooks = new(sync.Map)
}
if h, ok := Load(Hook); ok {
switch f := h.(type) {
case func(func(string, any), string, any):
f(handle, name, hook)
case func(string, any):
f(name, hook)
case func(func(string, any)):
f(handle)
default:
panic(Invalid)
}
} else {
handle(name, hook)
}
}
func Has(name string) (ok bool) {
_, ok = Load(name)
return ok
}
func List() []string {
return slices.Collect(ListSeq())
}
func ListSeq() iter.Seq[string] {
return func(yield func(string) bool) {
for name, _ := range ListSeq2() {
if !yield(name) {
break
}
}
}
}
func ListSeq2() iter.Seq2[string, any] {
return func(yield func(string, any) bool) {
if !Enabled() {
return
}
hooks.Range(func(name any, hook any) bool {
return yield(name.(string), hook)
})
}
}
func Load(name string) (hook any, ok bool) {
if Enabled() {
hook, ok = hooks.Load(name)
}
return hook, ok
}
- Go to the folder go/src/os and change the file
go/src/os/path.go
import (
"audit"
"internal/filepathlite"
"syscall"
)
...
func RemoveAll(path string) error {
if audit.Enabled() {
if h, ok := audit.Load(audit.RemoveAll); ok {
switch f := h.(type) {
case func(func(string) error, string) error:
return f(removeAll, path)
case func(string) error:
return f(path)
case func(func(string) error) error:
return f(removeAll)
default:
panic(audit.Invalid)
}
}
}
return removeAll(path)
}
...
- Create some test module with the file
main.go
package main
import (
"audit"
"fmt"
"os"
)
func main() {
if audit.Enabled() {
panic(fmt.Sprintf("Some third-party hooks: %v", audit.List()))
}
audit.Handle(audit.Hook, func(base func(string, any), name string, hook any) {
fmt.Println("Log Hook:", name)
base(name, hook)
})
audit.Handle(audit.RemoveAll, func(base func(string) error, path string) error {
fmt.Println("Log RemoveAll:", path)
return nil
})
os.RemoveAll("/")
}
It's all very simple and it works :) I think it's very easy to fully implement this, and it will be another unique feature in Go. Yes, it won't help from all attacks, but the main thing is that it will be a simple and at the same time powerful logging and monitoring tool.
Comment From: mwriter
Just in case, I tried another implementation based on pure functions
package audit
import (
"iter"
"slices"
"sync"
)
const (
Invalid = "Invalid audit hook"
)
var (
hooks *sync.Map
)
func Enabled() bool {
return hooks != nil
}
func Get(base any) (hook any) {
if h, ok := Load(base); ok {
hook = h
}
return hook
}
func handle(base any, hook any) {
hooks.Store(base, hook)
}
func Handle(base any, hook any) {
if !Enabled() {
hooks = new(sync.Map)
}
if h, ok := Load(Handle); ok {
switch f := h.(type) {
case func(func(any, any), any, any):
f(handle, base, hook)
default:
panic(Invalid)
}
} else {
handle(base, hook)
}
}
func Has(base any) (ok bool) {
_, ok = Load(base)
return ok
}
func List() []any {
return slices.Collect(ListSeq())
}
func ListSeq() iter.Seq[any] {
return func(yield func(any) bool) {
for base, _ := range ListSeq2() {
if !yield(base) {
break
}
}
}
}
func ListSeq2() iter.Seq2[any, any] {
return func(yield func(any, any) bool) {
if !Enabled() {
return
}
hooks.Range(func(base any, hook any) bool {
return yield(base, hook)
})
}
}
func Load(base any) (hook any, ok bool) {
if Enabled() {
hook, ok = hooks.Load(base)
}
return hook, ok
}
func RemoveAll(path string) error {
if audit.Enabled() {
if h, ok := audit.Load(RemoveAll); ok {
switch f := h.(type) {
case func(func(string) error, string) error:
return f(removeAll, path)
case func(string) error:
return f(path)
case func(func(string) error) error:
return f(removeAll)
default:
panic(audit.Invalid)
}
}
}
return removeAll(path)
}
func main() {
if audit.Enabled() {
panic(fmt.Sprintf("Some third-party hooks: %v", audit.List()))
}
audit.Handle(audit.Handle, func(handle func(any, any), base any, hook any) {
fmt.Println("Log Hook")
handle(base, hook)
})
audit.Handle(os.RemoveAll, func(base func(string) error, path string) error {
fmt.Println("Log RemoveAll:", path)
return nil
})
os.RemoveAll("/")
}
But of course predictably got an unsolvable problem :(
panic: runtime error: hash of unhashable type func(interface {}, interface {})
goroutine 1 [running]:
internal/sync.(*HashTrieMap[...]).Load(...)
internal/sync/hashtriemap.go:66 +0x5d
sync.(*Map).Load(...)
sync/hashtriemap.go:50
audit.Load(...)
audit/audit.go:84 +0x45
audit.Handle(...)
audit/audit.go:36 +0x72
Comment From: mwriter
New version based on functions as keys
package audit
import (
"iter"
"reflect"
"runtime"
"slices"
"sync"
)
const (
Invalid = "Invalid audit hook"
)
var (
bases *sync.Map
hooks *sync.Map
)
func hash(f any) uintptr {
defer func() {
recover()
}()
return reflect.ValueOf(f).Pointer()
}
func add(base any, hook any) {
h := hash(base)
if h == 0 {
return
}
bases.Store(h, base)
hooks.Store(h, hook)
}
func Add(base any, hook any) {
Enable()
if h, ok := Load(Add); ok {
switch f := h.(type) {
case func(func(any, any), any, any):
f(add, base, hook)
default:
panic(Invalid)
}
} else {
add(base, hook)
}
}
func clear() {
bases.Clear()
hooks.Clear()
}
func Clear() {
if !Enabled() {
return
}
if h, ok := Load(Clear); ok {
switch f := h.(type) {
case func(func()):
f(clear)
default:
panic(Invalid)
}
} else {
clear()
}
}
func delete(base any) {
h := hash(base)
if h == 0 {
return
}
bases.Delete(h)
hooks.Delete(h)
}
func Delete(base any) {
if !Enabled() {
return
}
if h, ok := Load(Delete); ok {
switch f := h.(type) {
case func(func(any), any):
f(delete, base)
default:
panic(Invalid)
}
} else {
delete(base)
}
}
func disable() {
clear()
bases = nil
hooks = nil
}
func Disable() {
if h, ok := Load(Disable); ok {
switch f := h.(type) {
case func(func()):
f(disable)
default:
panic(Invalid)
}
} else {
disable()
}
}
func Enable() {
if Enabled() {
return
}
bases = new(sync.Map)
hooks = new(sync.Map)
}
func Enabled() bool {
return hooks != nil
}
func Get(base any) (hook any) {
if h, ok := Load(base); ok {
hook = h
}
return hook
}
func Has(base any) (ok bool) {
_, ok = Load(base)
return ok
}
func List() []any {
return slices.Collect(ListSeq())
}
func ListSeq() iter.Seq[any] {
return func(yield func(any) bool) {
for base, _ := range ListSeq2() {
if !yield(base) {
break
}
}
}
}
func ListSeq2() iter.Seq2[any, any] {
return func(yield func(any, any) bool) {
if !Enabled() {
return
}
hooks.Range(func(base any, hook any) bool {
return yield(base, hook)
})
}
}
func ListNames() []string {
return slices.Collect(ListNamesSeq())
}
func ListNamesSeq() iter.Seq[string] {
return func(yield func(string) bool) {
if !Enabled() {
return
}
bases.Range(func(h any, base any) bool {
return yield(Name(base))
})
}
}
func Load(base any) (hook any, ok bool) {
if Enabled() {
hook, ok = hooks.Load(hash(base))
}
return hook, ok
}
func Name(f any) string {
return runtime.FuncForPC(hash(f)).Name()
}
func RemoveAll(path string) error {
if audit.Enabled() {
if h, ok := audit.Load(RemoveAll); ok {
switch f := h.(type) {
case func(func(string) error, string) error:
return f(removeAll, path)
case func(string) error:
return f(path)
case func(func(string) error) error:
return f(removeAll)
default:
panic(audit.Invalid)
}
}
}
return removeAll(path)
}
package main
import (
"audit"
"fmt"
"os"
)
func main() {
if audit.Enabled() {
panic(fmt.Sprintf("Some third-party hooks: %v", audit.ListNames()))
}
audit.Add(os.RemoveAll, func(base func(string) error, path string) error {
fmt.Println("Log RemoveAll:", path)
return nil
})
audit.Add(audit.Delete, func(handle func(any), base any) {
fmt.Println("Log DeleteHook:", audit.Name(base))
})
audit.Add(audit.Clear, func(handle func()) {
fmt.Println("Log ClearHooks")
})
audit.Add(audit.Disable, func(handle func()) {
fmt.Println("Log DisableAudit")
})
audit.Add(audit.Add, func(handle func(any, any), base any, hook any) {
fmt.Println("Log AddHook:", audit.Name(base))
})
malicious()
}
func malicious() {
audit.Delete(os.RemoveAll)
audit.Add(os.RemoveAll, func(base func(string) error, path string) error {
return base(path)
})
audit.Clear()
audit.Disable()
os.RemoveAll("/")
}