Proposal Details
As recommended by Joe Tsai (https://github.com/go-json-experiment/json/issues/168), I am opening this proposal.
The current v2 JSON API allows for customized generic marshalers and unmarshalers, such as:
// MarshalFunc constructs a type-specific marshaler that
// specifies how to marshal values of type T.
func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers
I attempted to adapt my enum package package to the new v2 API in order to support interface-based marshalling and unmarshalling using various styles (e.g., internally or externally tagged). However, I encountered a limitation: the current API only supports generic type parameters for defining custom marshalers and unmarshalers. Since my implementation relies on dynamic type discovery via reflect.Type, I cannot instantiate the required generic types.
To address this, I propose extending the API with additional functions that accept reflect.Type, for example:
// MarshalFuncOf constructs a type-specific marshaler that
// specifies how to marshal values of the given reflect.Type.
func MarshalFuncOf(t reflect.Type, fn func(reflect.Type) ([]byte, error)) *Marshalers
// UnmarshalFuncOf constructs a type-specific unmarshaler that
// specifies how to unmarshal values of the given reflect.Type.
func UnmarshalFuncOf(t reflect.Type, fn func([]byte, reflect.Type) error) *Unmarshalers
This approach would be consistent with how the reflect package provides both TypeOf(any) and TypeFor[T], enabling both static and dynamic use cases.
Alternatively, or in addition, I would also welcome native support in the v2 package for interface-based marshalling and unmarshalling, which would significantly improve flexibility for libraries relying on dynamic types.
Comment From: gabyhelp
Related Issues
- proposal: encoding/json support Unmarshal type interface #23549 (closed)
- proposal: encoding/json: Unmarshal support for dynamic types #13338 (closed)
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
Comment From: majewsky
func MarshalFuncOf(t reflect.Type, fn func(reflect.Type) ([]byte, error)) *Marshalers
Is the second reflect.Type a typo? I feel like you mean to take reflect.Value there. (Same on UnmarshalFuncOf.)
Comment From: torbenschinke
Thanks for reviewing. Yes, you are right, my API proposal was a bit hasty. However, just replacing reflect.Type with reflect.Value won't work when encoding or decoding into an interface type, which is my primary goal.
The registered Marshaler must know the required interface type to decide how to encode an actual value. The information about the interface may have been lost, once it passed through any, so I would carry it independently. An actual type may satisfy multiple interfaces which may need different encodings (e.g. in one interface with adjacent and in another with internally tagged discriminators).
The same is true for UnmarshalFuncOf, which also requires the interface type to decide on the proper decoding strategy.
I propose to let API users to work with any when accepting or returning concrete values, to make implementations clearer and to minimize the usage of reflection where possible. But I would be also fine to use a reflect.Value:
// MarshalFuncOf constructs a type-specific marshaler that
// specifies how to marshal values of the given reflect.Type. The marshal func accepts the registered
// type and an actual value whose type is either equal or assignable to t.
func MarshalFuncOf(t reflect.Type, fn func(reflect.Type, any) ([]byte, error)) *Marshalers
// UnmarshalFuncOf constructs a type-specific unmarshaler that
// specifies how to unmarshal values of the given reflect.Type. The unmarshal func accepts the registered
// type and returns a new instance whose type is either equal or assignable to t.
func UnmarshalFuncOf(t reflect.Type, fn func([]byte, reflect.Type) (any,error)) *Unmarshalers
Update: fixed typos
Comment From: dsnet
As an implementation note, this unfortunately will not perform well. One of the reasons we use type parameters is that we can call the user provided function directly, and the compiler can allocate all the relevant input/return arguments on the stack. Working with just an reflect.Type means that we will need to use reflect.Value.Call, which is relatively slow and heap allocation heavy.
Comment From: torbenschinke
My original intention was to develop a developer-friendly API to represent common types of polymorphic encoding and decoding. Based on this, I tried to adapt the existing generic pattern. To improve performance, we could introduce a more specific API for this purpose.
Is a more specific API actually an option?
Comment From: tie
Working with just an reflect.Type means that we will need to use reflect.Value.Call, which is relatively slow and heap allocation heavy.
@dsnet, I don’t see why we need to use reflect.Value.Call for this proposal. I have a similar use case and the patch below should suffice:
diff --git a/arshal_funcs.go b/arshal_funcs.go
index 0b7e82f..9122f9b 100644
--- a/arshal_funcs.go
+++ b/arshal_funcs.go
@@ -171,11 +171,17 @@ func (a *typedArshalers[Coder]) lookup(fnc func(*Coder, addressableValue, *jsono
// It may not return [SkipFunc].
func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers {
t := reflect.TypeFor[T]()
+ return MarshalFuncOf(t, func(v reflect.Value) ([]byte, error) {
+ return fn(v.Interface().(T))
+ })
+}
+
+func MarshalFuncOf(t reflect.Type, fn func(reflect.Value) ([]byte, error)) *Marshalers {
assertCastableTo(t, true)
typFnc := typedMarshaler{
typ: t,
fnc: func(enc *jsontext.Encoder, va addressableValue, mo *jsonopts.Struct) error {
- val, err := fn(va.castTo(t).Interface().(T))
+ val, err := fn(va.castTo(t))
if err != nil {
err = wrapSkipFunc(err, "marshal function of type func(T) ([]byte, error)")
if mo.Flags.Get(jsonflags.ReportErrorsWithLegacySemantics) {
@@ -213,6 +219,12 @@ func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers {
// must not be retained outside the function call.
func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers {
t := reflect.TypeFor[T]()
+ return MarshalToFuncOf(t, func(enc *jsontext.Encoder, v reflect.Value) error {
+ return fn(enc, v.Interface().(T))
+ })
+}
+
+func MarshalToFuncOf(t reflect.Type, fn func(*jsontext.Encoder, reflect.Value) error) *Marshalers {
assertCastableTo(t, true)
typFnc := typedMarshaler{
typ: t,
@@ -220,7 +232,7 @@ func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers {
xe := export.Encoder(enc)
prevDepth, prevLength := xe.Tokens.DepthLength()
xe.Flags.Set(jsonflags.WithinArshalCall | 1)
- err := fn(enc, va.castTo(t).Interface().(T))
+ err := fn(enc, va.castTo(t))
xe.Flags.Set(jsonflags.WithinArshalCall | 0)
currDepth, currLength := xe.Tokens.DepthLength()
if err == nil && (prevDepth != currDepth || prevLength+1 != currLength) {
@@ -259,6 +271,12 @@ func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers {
// It may not return [SkipFunc].
func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers {
t := reflect.TypeFor[T]()
+ return UnmarshalFuncOf(t, func(buf []byte, v reflect.Value) error {
+ return fn(buf, v.Interface().(T))
+ })
+}
+
+func UnmarshalFuncOf(t reflect.Type, fn func([]byte, reflect.Value) error) *Unmarshalers {
assertCastableTo(t, false)
typFnc := typedUnmarshaler{
typ: t,
@@ -267,7 +285,7 @@ func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers {
if err != nil {
return err // must be a syntactic or I/O error
}
- err = fn(val, va.castTo(t).Interface().(T))
+ err = fn(val, va.castTo(t))
if err != nil {
err = wrapSkipFunc(err, "unmarshal function of type func([]byte, T) error")
if uo.Flags.Get(jsonflags.ReportErrorsWithLegacySemantics) {
@@ -295,6 +313,12 @@ func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers {
// must not be retained outside the function call.
func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers {
t := reflect.TypeFor[T]()
+ return UnmarshalFromFuncOf(t, func(dec *jsontext.Decoder, v reflect.Value) error {
+ return fn(dec, v.Interface().(T))
+ })
+}
+
+func UnmarshalFromFuncOf(t reflect.Type, fn func(*jsontext.Decoder, reflect.Value) error) *Unmarshalers {
assertCastableTo(t, false)
typFnc := typedUnmarshaler{
typ: t,
@@ -302,7 +326,7 @@ func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers
xd := export.Decoder(dec)
prevDepth, prevLength := xd.Tokens.DepthLength()
xd.Flags.Set(jsonflags.WithinArshalCall | 1)
- err := fn(dec, va.castTo(t).Interface().(T))
+ err := fn(dec, va.castTo(t))
xd.Flags.Set(jsonflags.WithinArshalCall | 0)
currDepth, currLength := xd.Tokens.DepthLength()
if err == nil && (prevDepth != currDepth || prevLength+1 != currLength) {
I.e. we can factor out reflect.TypeFor[T] and v.Interface().(T) type cast into the function with type parameters, and the rest of the implementation already relies on reflect.Type & reflect.Value.
func XXXFunc[T any](fn func(T) ([]byte, error)) *XXXarshalers {
t := reflect.TypeFor[T]()
return XXXFuncOf(t, func(v reflect.Value) ([]byte, error) {
return fn(v.Interface().(T))
})
}
func XXXFuncOf(t reflect.Type, fn func(reflect.Value) ([]byte, error)) *XXXarshaler {
// Existing implementation, but without `v.Interface().(T)` type cast.
}
Comment From: dsnet
Thank you @tie for the example. You are correct. I misremembered the original function call signature and that the user provided function doesn't contain a concrete type, but rather takes in a reflect.Value (the original post has a typo as it says reflect.Type instead).
To be exact, this proposal would be adding API like:
func MarshalFuncOf(t reflect.Type, fn func(reflect.Value) ([]byte, error)) *Marshalers
func MarshalToFuncOf(t reflect.Type, fn func(*jsontext.Encoder, reflect.Value) error) *Marshalers
func UnmarshalFuncOf(t reflect.Type, fn func([]byte, reflect.Value) error) *Unmarshalers
func UnmarshalFromFuncOf(t reflect.Type, fn func(*jsontext.Decoder, reflect.Value) error) *Unmarshalers
This is a notable amount of API surface to add, but fortunately go/doc would sort them under the Marshalers and Unmarshalers type, respectively.