With GOEXPERIMENT=jsonv2, the behavior of encoding/json.Decoder.Token has changed when encountering an unexpected EOF.

        text := `{`
        dec := json.NewDecoder(strings.NewReader(text))
        for {
                t, err := dec.Token()
                fmt.Printf("%v %v\n", t, err)
                if err != nil {
                        break
                }
        }

v1:

{ <nil>
<nil> EOF

v2:

{ <nil>
<nil> unexpected end of JSON input

I haven't exhaustively tested this, but the v1 Decoder appears to consistently return io.EOF when encountering EOF, even when this happens in the middle of an object: {, {"a", {"a":, etc.

The documentation is a bit ambiguous: "At the end of the input stream, Token returns nil, io.EOF.", but also "Token guarantees that the delimiters [ ] { } it returns are properly nested and matched". It appears that the first sentence takes precedence over the second, and the caller of Token is responsible for verifying that EOF happens at an expected position.

Comment From: dsnet

I'm reminded of #69449, where I argued that the v2 jsontext.Decoder.Readtoken behavior is correct and that v1 is incorrect. That said, with the upcoming possible release of "jsontext" it probably makes sense to keep existing v1 behavior if possible and have jsontext.Decoder.ReadToken be the proper behavior.

Comment From: neild

Indeed, this appears to be json/v2 inadvertently(?) fixing #69449.

If we change this, we should probably change it consistently regardless of whether we're using the wrapped-v2 encoding/json.Decoder or not.

Comment From: gopherbot

Change https://go.dev/cl/689516 mentions this issue: encoding/json: fix truncated Token error regression in goexperiment.jsonv2

Comment From: gabyhelp

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Comment From: neild

FWIW, I'm pretty sure the failing test (google-internal) that led me to discover this should be failing--it's operating on an inadvertently invalid JSON chunk that's missing a closing }, but the system under test fails to detect the unbalanced brackets. If we fix #69449, that code will get better at validation.