Golang proposal: log/slog: native support for handling List

Proposal Details

The current implementation of log/slog does not fully support logging arrays or slices. As a result, the type implementing the slog.Valuer interface is not respected when passed as an element of an array or slice and the Handler cannot handle the elements within the input array or slice either.

One possible workaround is to create a custom List using a generic struct that implements the slog.LogValuer interface, returns a Group and uses the index as the key. An example of this approach can be found in this comment https://github.com/golang/go/issues/63204#issuecomment-2428962553

However, it would make more sense and be more practical if List were natively supported

type listptr  *Value    // used in Value.any when the Value is a []Value

const (
    KindAny Kind = iota
    KindBool
    KindDuration
    KindFloat64
    KindInt64
    KindString
    KindTime
    KindUint64
    KindGroup
    KindLogValuer
        KindList 
)

// ListValue returns a [Value] for a slice of arbitrary values.
func ListValue(vs ...Value) Value {
    return Value{num: uint64(len(vs)), any: listptr(unsafe.SliceData(vs))}
}

// List returns v's value as a []Value.
// It panics if v's [Kind] is not [KindList].
func (v Value) List() []Value{
    if sp, ok := v.any.(listptr); ok {
        return unsafe.Slice((*Value)(sp), v.num)
    }
    panic("List: bad kind")
}

func (v Value) list() []Value{
    return unsafe.Slice((*Value)(v.any.(listptr)), v.num)
}

Comment From: ianlancetaylor

CC @jba

Comment From: jba

Can you provide more information?

• Examples of where this is useful.

• How a Handler would render a KindList. In particular the two build-in handlers.

• Suggestions for how to deal with the breakage this would cause for all existing handlers.

I'm mostly concerned with adding a low-value feature that breaks a lot of existing code.

Comment From: longit644

Hi @jba,

Thank you for your response. Here’s more detail on how this proposal would be useful, especially in the context of how existing handlers would work with a KindList and the associated impact.

This proposal also addresses issue https://github.com/golang/go/issues/62699, independent of any specific format like JSON. By introducing KindList, we enable logging for slices or arrays without being tied to a single format such as JSON. This flexibility allows users to implement handlers for alternative formats, such as CBOR or MsgPack. As a result, the solution becomes more adaptable and extensible, catering to a broader range of use cases. This empowers users to easily customize the logging output according to their unique requirements.

I believe that KindList will be a great building block for log/slog, filling the missing piece for logging slices and arrays - fundamental data structures.

Examples of Where This Is Useful

This proposal is particularly useful in scenarios where we need to log sensitive or redacted data in structured logging systems, especially when the data is part of collections (e.g., slices, arrays). For example:

Redacting Sensitive Information in Lists

// A token is a secret value that grants permissions.
type Token string

// It avoids revealing the token.
func (Token) LogValue() slog.Value {
    return slog.StringValue("[REDACTED]")
}

func main() {
    tokens := []Token{"keep it secret", "keep it safe"}
    logger.Info("Permission revoked", slog.List("tokens", tokens...))
    // Or
    logger.Info("Permission revoked", "tokens", slog.ListValue(
        func(tokens []Token) []slog.Value {...}()..., // Transform tokens to slog.Values
    ))
}

Redacting Personally Identifiable Information (PII) Similarly, for more complex types like PhoneNumber or Address, the proposal allows for automatic redaction when logging lists of users.

type PhoneNumberType int

const (
    PhoneNumberTypeUnspecified = iota
    PhoneNumberTypeMobile
    PhoneNumberTypeHome
    PhoneNumberTypeWork
)

// Struct for phone numbers
type PhoneNumber struct {
    Type   PhoneNumberType
    Number string
}

// It avoids revealing the phone number.
func (p PhoneNumber) LogValue() slog.Value {
    return slog.GroupValue(
        slog.Int("type", p.Type),
        slog.String("number", "[REDACTED]"),
    )
}

type AddressType int

const (
    AddressTypeUnspecified = iota
    AddressTypeHome
    AddressTypeOffice
)

type Address struct {
    Type    AddressType
    Address string
}

// It avoids revealing the address.
func (a Address) LogValue() slog.Value {
    return slog.GroupValue(
        slog.Int("type", a.Type),
        slog.String("address", "[REDACTED]"),
    )
}

// Struct for User
type User struct {
    ID           int
    Name         string
    Email        string
    PhoneNumbers []PhoneNumber
    Addresses    []Address
}

// It avoids revealing the PII user data.
func (u User) LogValue() slog.Value {
    return slog.GroupValue(
        slog.Int("id", u.ID),
        slog.String("name", "[REDACTED]"),
        slog.String("email", mask.Email(u.Email)),
        slog.List("phone_numbers", u.PhoneNumbers...),
        slog.List("addresses", u.Addresses...),
    )
}

func main() {
    user := User{
        ID:           1,
        Name:         "Nguyen Van A",
        Email:        "a.nguyen@example.com",
        PhoneNumbers: []PhoneNumber{{Type: PhoneNumberTypeMobile, Number: "123-456-7890"}},
        Addresses:    []Address{{Type: AddressTypeHome, Address: "123 Hoang Dieu St"}},
    }
    logger.Info("User profile updated", "user", user)
}

How a Handler Would Render a KindList, In Particular the Two Built-In Handlers

TextHandler With TextHandler, the logged list will print with the index in logging key, like this:

logger.Info("Permission revoked", slog.List("tokens", tokens...))

Will print as:

token[0]=[REDACTED] token[1]=[REDACTED]

JSONHandler With JSONHandler, the list will be rendered as a normal JSON array:

logger.Info("Permission revoked", slog.List("tokens", tokens...))

Will print as:

tokens: ["[REDACTED]", "[REDACTED]"]

Suggestions for How to Deal with the Breakage This Would Cause for All Existing Handlers

I understand the concern about potential breakage, as many existing handlers may not expect a KindList and could lead to runtime errors or inconsistent outputs if the data structures aren’t handled correctly. However, by introducing a new type (which has not been used before by any users), we can ensure that the existing built-in or community handlers will not be affected by the change.

When this feature is released, the supported version of the built-in handlers will be shipped together. In the meantime, users can wait for community handlers to be updated before using this feature.

Let me know if you need further clarification or have any other concerns!

Comment From: jba

community handlers will not be affected by the change

I'm not sure what makes these special so that they are immune to the change. They could break too.

Also, your examples don't compile with your code, since the slog.ListValue function takes slog.Value, not any.

But maybe that's a good thing, because it suggests a simpler solution. What if the built-in handlers treated []Value specially, by converting it to []any with v.Resolve().Any() for each value in the slice? I said elsewhere that it's infeasible to walk every slice, but this is confined to []Value, which users have to go out of their way to create.

We could make this an "official" feature of slog.Handler by adding a check in testing/slogtest, so that other handlers would realize that they should add it (or explicitly opt out of doing so).

Now nothing will break, and the ecosystem will gradually improve in this direction.

Comment From: jaloren

@jba as luck would have it, I recently encountered this limitation and I can provide an experience report for why this feature would be nice to have. I am a SRE who uses go to write various orchestration tools. These tools need to interact with various different systems, where each system produces a stream of events into their logs. These events are semi-structured where some events are json documents and others are strings.

We strongly believe in structured logging where ever possible and default to json since its a common denominator that just about any system can parse reliably. As a consequence, the orchestration tool uses the slog JSON handler. When the tool encounters a failure with another system, the tool collects any events from the system and log them into slog as a list of events where the key is "events" and the value is a slice of Events. The slice of events will have polymorphic data types where each event can either be a string or a json document. I don't want the json documents to be rendered as strings because that's really hard for people to read and much harder for logging systems to parse.

I ended up with this:

https://go.dev/play/p/c5EoOgj4N73

I suspect your proposal for passing in a slice of values would work fine.

Comment From: seankhliao

Glancing through existing handler implementations, I don't think it's safe to add new Kinds, handlers may panic, add prefixes indicating it's unhandled, or silently drop the field. https://github.com/search?q=language%3AGo+%2Fcase+slog.kind%2F&type=code

Comment From: longit644

Hi @seankhliao, @jba

Thank you for your response!

Glancing through existing handler implementations, I don't think it's safe to add new Kinds, handlers may panic, add prefixes indicating it's unhandled, or silently drop the field. https://github.com/search?q=language%3AGo+%2Fcase+slog.kind%2F&type=code

I'm not sure what makes these special so that they are immune to the change. They could break too.

I share your concern, and I’ve carefully considered this as well. The potential for breaking changes only arises once users begin using the new List feature. Until then, the existing user code base should remain unaffected.

However, when users start using List, they will need to update their handlers to correctly handle the new Kind. If they don’t, the handlers may panic, drop fields, or behave unpredictably.

Comment From: seankhliao

The design of slog is to allow for the separation of the logging frontend (slog.Logger) and backend (slog.Handler). A third party package may accept being configured with a slog.Logger without needing to know what the actual slog.Handler is. In this case, you may not be aware of the need to update your handler to support the new kind until your application crashes because a package you didn't write but just upgraded used the new feature.

Comment From: longit644

Hi everyone, apologies for my long absence over the past few months. Thank you all for your thoughtful comments and feedback on the earlier version of this proposal.

After carefully considering the design and incorporating the suggestion from @jba, I now propose a simplified and backward-compatible approach to add native list support to slog. This leverages the existing KindAny and extends testing/slogtest, without introducing a new Kind.

Introduce New APIs

Constructors

We introduce functions to create a Value or Attr from a list:

type listptr *Value // used in Value.any when the Value is a []Value

// ListValue returns a Value representing a list of arbitrary values.
func ListValue[T any](vs ...T) Value {
    switch vs2 := any(vs).(type) {
    case []Value:
        return Value{num: uint64(len(vs2)), any: listptr(unsafe.SliceData(vs2))}
    default:
        return Value{
            num: uint64(len(vs)),
            any: listptr(unsafe.SliceData(argsToValueSlice(vs))),
        }
    }
}

// List creates an Attr whose value is a list of Values.
func List[T any](key string, args ...T) Attr {
    return Attr{key, ListValue(args...)}
}

func argsToValueSlice[T any](args []T) []Value {
    vs := make([]Value, 0, len(args))
    for _, arg := range args {
        vs = append(vs, AnyValue(arg))
    }
    return vs
}

Accessors

Handlers can extract the list with:

// List returns v's value as a []Value.
// Panics if the value is not a list.
func (v Value) List() []Value {
    if sp, ok := v.any.(listptr); ok {
        return unsafe.Slice((*Value)(sp), v.num)
    }
    panic("List: not a list value")
}

func (v Value) list() []Value {
    return unsafe.Slice((*Value)(v.any.(listptr)), v.num)
}

Helper

Because we introduce a new type of container (List) without adding a new Kind, these helper methods are useful for handlers to distinguish between container types:

// IsGroup reports whether the value is a group.
func (v Value) IsGroup() bool {
    _, ok := v.any.(groupptr)
    return ok
}

// IsList reports whether the value is a list.
func (v Value) IsList() bool {
    _, ok := v.any.(listptr)
    return ok
}

Update Existing APIs

Accessors

We update Value.Any() to support lists:

// Any returns v's value as an any.
func (v Value) Any() any {
    switch v.Kind() {
    case KindAny:
        switch vt := v.any.(type) {
        case kind:
            return Kind(vt)
        case listptr:
            return v.list()
        default:
            return v.any
        }
    // ...
    }
}

Other

Add support for comparing list values:

// Equal reports whether v and w represent the same Go value.
func (v Value) Equal(w Value) bool {
    k1 := v.Kind()
    k2 := w.Kind()
    if k1 != k2 {
        return false
    }
    switch k1 {
    case KindInt64, KindUint64, KindBool, KindDuration:
        return v.num == w.num
    case KindString:
        return v.str() == w.str()
    case KindFloat64:
        return v.float() == w.float()
    case KindTime:
        return v.time().Equal(w.time())
    case KindAny, KindLogValuer:
        if v.IsList() && w.IsList() {
            return slices.EqualFunc(v.list(), w.list(), Value.Equal)
        }

        return v.any == w.any // may panic if non-comparable
    case KindGroup:
        return slices.EqualFunc(v.group(), w.group(), Attr.Equal)
    default:
        panic(fmt.Sprintf("bad kind: %s", k1))
    }
}

Append text format of list to bytes.

// append appends a text representation of v to dst.
// v is formatted as with fmt.Sprint.
func (v Value) append(dst []byte) []byte {
    switch v.Kind() {
    case KindString:
        return append(dst, v.str()...)
    case KindInt64:
        return strconv.AppendInt(dst, int64(v.num), 10)
    case KindUint64:
        return strconv.AppendUint(dst, v.num, 10)
    case KindFloat64:
        return strconv.AppendFloat(dst, v.float(), 'g', -1, 64)
    case KindBool:
        return strconv.AppendBool(dst, v.bool())
    case KindDuration:
        return append(dst, v.duration().String()...)
    case KindTime:
        return append(dst, v.time().String()...)
    case KindGroup:
        return fmt.Append(dst, v.group())
    case KindAny, KindLogValuer:
        if _, ok := v.any.(listptr); ok {
            return fmt.Append(dst, v.list())
        }

        return fmt.Append(dst, v.any)
    default:
        panic(fmt.Sprintf("bad kind: %s", v.Kind()))
    }
}

ReplaceAttr

Since we’re adding a new container type (List), we replace groups []string with a more general path []string in ReplaceAttr:

// ReplaceAttr is called to rewrite each non-group attribute before it is
// logged. The attribute's value has been resolved (see [Value.Resolve]).
// If ReplaceAttr returns a zero Attr, the attribute is discarded.
//
// The built-in attributes with keys "time", "level", "source", and "msg"
// are passed to this function, except that time is omitted
// if zero, and source is omitted if AddSource is false.
//
// The first argument is the path to the current group that contains the
// Attr. It must not be retained or modified. ReplaceAttr is never called
// for Group attributes, only their contents. For example, the attribute
// list
//
//     Int("a", 1), Group("g", Int("b", 2)), Int("c", 3), List("l", GroupValue(Int("d", 4)))
//
// results in consecutive calls to ReplaceAttr with the following arguments:
//
//     nil, Int("a", 1)
//     []string{"g"}, Int("b", 2)
//     nil, Int("c", 3)
//     []string{"l", "[0]"}, Int("d", 4)
//
// ReplaceAttr can be used to change the default keys of the built-in
// attributes, convert types (for example, to replace a `time.Time` with the
// integer seconds since the Unix epoch), sanitize personal information, or
// remove attributes from the output.
ReplaceAttr func(path []string, a Attr) Attr

Example and output format

This example demonstrates how to log redacted tokens using both a plain slice and a type that implements slog.LogValuer.

// A token is a secret value that grants permissions.
type Token string

// LogValue implements slog.LogValuer.
// It avoids revealing the token.
func (Token) LogValue() slog.Value {
    return slog.StringValue("REDACTED_TOKEN")
}

// Tokens represents a list of tokens.
type Tokens []Token

// LogValue implements slog.LogValuer.
// It avoids revealing the actual tokens.
func (ts Tokens) LogValue() slog.Value {
    return slog.ListValue(ts...)
}

// This example demonstrates a Value that replaces itself
// with an alternative representation to avoid revealing secrets.
func ExampleLogValuer_secret() {
    t := Token("shhhh!")
    logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: slogtest.RemoveTime}))
    logger.Info("permission granted", "user", "Perry", "token", t)

    ts1 := []Token{"shhhh!", "whissss!"}
    logger.Info("user session info", "user", "Perry", "tokens", slog.ListValue(ts1...))

    ts2 := Tokens{"zhhhh!", "whizzzz!"}
    logger.Info("user session info", "user", "Heinz", "tokens", ts2)
}

Output

JSON format:

{"level":"INFO","msg":"permission granted","user":"Perry","token":"REDACTED_TOKEN"}
{"level":"INFO","msg":"user session info","user":"Perry","tokens":["REDACTED_TOKEN","REDACTED_TOKEN"]}
{"level":"INFO","msg":"user session info","user":"Heinz","tokens":["REDACTED_TOKEN","REDACTED_TOKEN"]}

Text format:

level=INFO msg="permission granted" user=Perry token=REDACTED_TOKEN
level=INFO msg="user session info" user=Perry tokens[0]=REDACTED_TOKEN tokens[1]=REDACTED_TOKEN
level=INFO msg="user session info" user=Heinz tokens[0]=REDACTED_TOKEN tokens[1]=REDACTED_TOKEN

Reference Implementation

You can find a working reference implementation with examples here: https://github.com/longit644/go/commit/b0de3ae198a03db8391f3d835ef52817ca0653b8

Comment From: longit644

Hi @jba, @seankhliao. Please take a look on this when you have moment. Thank you.

Comment From: jba

What's wrong with my suggestion on https://github.com/golang/go/issues/71088#issuecomment-2569217300? That requires no new API and only affects handlers. Can you describe what your recent proposal adds that would be better than just treating []Value specially?