Golang encoding/binary: add func Append - Aurora Blog|java/go/python

I'd like to propose adding a function with the following signature:

// Append the binary representation of data to buf.
//
// buf may be nil, in which case a new buffer will be allocated. See [Write] on
// which data are acceptable.
//
// Returns the (possibly extended) buffer containing data or an error.
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

This is useful when repeatedly encoding the same kind of value multiple times into a larger buffer and is a natural extension to #50601. A related proposal wants to add similar functions to other packages in encoding: #53693.

Together with #53685 it becomes possible to implement a version of binary.Write that doesn't allocate when using common io.Writer. See my comment for writeBuffer(). Roughly (untested):

func write(w io.Writer, order binary.AppendByteOrder, data any) error {
    buf := writeBuffer(w, binary.Size(data))
    binary.Append(buf[:0], order, data)
    _, err := w.Write(buf)
   return err
}

If the CLs to avoid escaping in reflect APIs lands, Append would allow encoding with zero allocations.

I think it might also allow encoding into stack allocated slices, provided the compiler is (or becomes) smart enough:

buf := binary.Append(make([]byte, 0, 128), binary.LittleEndian, v)

Comment From: ianlancetaylor

CC @dsnet

Comment From: dsnet

SGTM. Write is the last API in binary that lacks an append-like equivalent.

Comment From: lmb

I've been playing with the implementation a bit, and I'd like to extend my proposal to cover an equivalent function for Read.

// Decode data from buf according to the given byte order.
//
// Returns the number of bytes read from the beginning of buf or io.ErrUnexpectedEOF.
func Decode(buf []byte, order ByteOrder, data any) (int, error)

The rationale is similar to Append: using this new function makes it possible to implement zero-allocation decode for common io.Reader.

I struggled to come up with a good name for the function, what is the inverse of Append? Renaming it to Encode seems clearer to me, and follows existing unexported types. The proposal would become:

func Decode(buf []byte, order ByteOrder, data any) (int, error) // aka Read
func Encode(buf []byte, order AppendByteOrder, data any) ([]byte, error) // aka Write

Comment From: rsc

We have used Append as the word for all of the other appending encoders, both in this package and others; it is confusing to change to Encode now (for example, why does Encode take an AppendByteOrder and not an EncodeByteOrder? why is it Encode but AppendVarint? and so on). Let's keep using Append.

Comment From: rsc

If we need the decoding version, it could be Parse.

Comment From: rsc

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings. — rsc for the proposal review group

Comment From: rsc

Looking at unicode/utf8, which has both AppendRune and DecodeRune, it would be fine to have Decode+Append here. But sometimes you are not actually appending, and it seems okay to add Encode too, like utf8 has EncodeRune. But Encode would not be an appender. It would return an error if the buffer was not large enough.

If we did that, we'd have all three:

func Decode(buf []byte, order ByteOrder, data any) (int, error) 
func Encode(buf []byte, order ByteOrder, data any) (int, error)
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

That would match utf8 better.

Comment From: gopherbot

Change https://go.dev/cl/579157 mentions this issue: encoding/binary: add Append, Encode and Decode

Comment From: lmb

I've dusted off the code I've had lying around and mailed a CL. Encode and Decode were straight forward additions. One interesting wrinkle: I've opted to implement all other primitive functions in terms of Append, which means I need to go from ByteOrder to AppendByteOrder:

func toAppendByteOrder(order ByteOrder) AppendByteOrder {
    switch order := order.(type) {
    case littleEndian:
        return order
    case bigEndian:
        return order
    case nativeEndian:
        return order
    case AppendByteOrder:
        return order
    default:
        return appendableByteOrder{order}
    }
}

toAppendByteOrder can only be implemented this way in encoding/binary because the types are not exported. Doing a type switch vs comparing interface values (which is possible outside of the package) does make a difference in the benchmarks.

P.S.: ToAppendByteOrder is necessary if a dependent package takes ByteOrder as an argument but would like to use Append internally for performance reasons. Without it the choice of encoding primitive „leaks“ into the API and may require breaking changes. I’d like to extend the proposal to include the function (or I can make a separate proposal).

Comment From: rsc

Sorry but we're not going to add ToAppendByteOrder. It seems okay to have some duplication of special cases between Encode and Append.

Comment From: rsc

Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group

The proposal is to add:

func Decode(buf []byte, order ByteOrder, data any) (int, error) func Encode(buf []byte, order ByteOrder, data any) (int, error) func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

Comment From: rsc

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group

The proposal is to add:

Comment From: aclements

We've been going around and around on the implementation of this and I think I just realized the central issue that's giving us trouble: why does Append take an AppendByteOrder instead of a ByteOrder? In the general case, it already pre-computes the size of the entire encoded value and grows the slice by that much. At that point, there's no need for the Append* methods of AppendByteOrder. In the current CL, it uses the Append methods for the fast path, but I'm not sure there's much value in that over just growing the slice.

Comment From: rsc

Dropping AppendByteOrder works for me.

Comment From: gopherbot

Change https://go.dev/cl/587096 mentions this issue: encoding/binary: adjust docs for Append, Encode and Decode