Viper UnmarshalKey looses some data if a subitem is overridden

Expected behavior (what you expected to happen): viper.UnmarshalKey should unmarshals whole structure, not overridden keys only

Actual behavior (what actually happened): if viper.Set has been used to override a value of a sub item, then viper.UnmarshalKey unmarshals only overridden keys

Repl.it link:

https://replit.com/@SVilgelm/viperUnmarshalKey-bug

Code reproducing the issue:

package main

import (
    "strings"

    "github.com/spf13/viper"
)

func main() {
    config := `
resources:
  baz:
    name: "Super bazzz"
    xyz: true
    test: 123
`
    viper.Debug()
    println("########################################################################")

    viper.SetConfigType("yaml")
    viper.ReadConfig(strings.NewReader(config))
    viper.Debug()
    println("########################################################################")

    res1 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res1)
    println("res1 Name:", res1.Name, "; XYZ:", res1.XYZ)
    println("########################################################################")

    viper.Set("resources.baz.name", "override name")
    viper.Debug()
    println("########################################################################")

    res2 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res2)
    println("res2 Name:", res2.Name, "; XYZ:", res2.XYZ)
    println("########################################################################")
}

Environment: - Viper version:v1.7.1 - Config source: any - File format: any

Comment From: github-actions[bot]

👋 Thanks for reporting!

A maintainer will take a look at your issue shortly. 👀

In the meantime: We are working on Viper v2 and we would love to hear your thoughts about what you like or don't like about Viper, so we can improve or fix those issues.

⏰ If you have a couple minutes, please take some time and share your thoughts: https://forms.gle/R6faU74qPRPAzchZ9

📣 If you've already given us your feedback, you can still help by spreading the news, either by sharing the above link or telling people about this on Twitter:

https://twitter.com/sagikazarmark/status/1306904078967074816

Thank you! ❤️

Comment From: SVilgelm

Actually the issue with viper.Get, it does not return the whole structure

Comment From: madsoigard

any update on this?

Comment From: sagikazarmark

The problem is that resources.baz exists as a key, so UnmashalKey won't look any further. I wonder what happens if you create a Resources struct and try to unmarshal resources instead. I think that should work.

Comment From: SVilgelm

@sagikazarmark I have updated the repl, the problem still exists even if I unmarshal resources. I also updated th go.mod to use latest viper.

https://replit.com/@SVilgelm/viperUnmarshalKey-bug#main.go

Code:

package main

import (
    "strings"
  "fmt"

    "github.com/spf13/viper"
)

func main() {
    config := `
resources:
  baz:
    name: "Super bazzz"
    xyz: true
    test: 123
`
    viper.Debug()
    println("########################################################################")

    viper.SetConfigType("yaml")
    viper.ReadConfig(strings.NewReader(config))
    viper.Debug()
    println("########################################################################")

    res1 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res1)
    println("res1 Name:", res1.Name, "; XYZ:", res1.XYZ)
    println("########################################################################")

    viper.Set("resources.baz.name", "override name")
    viper.Debug()
    println("########################################################################")

    res2 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res2)
    println("res2 Name:", res2.Name, "; XYZ:", res2.XYZ)
    println("########################################################################")

  println("########################################################################")

  res3 := struct {
    Baz struct {
      Name string `mapstructure:"name"`
      XYZ  bool   `mapstructure:"xyz"`
    } `mapstructure:"baz"`
    }{}
    viper.UnmarshalKey("resources", &res3)
    println("res3 Name:", res3.Baz.Name, "; XYZ:", res3.Baz.XYZ)
    println("########################################################################")

  fmt.Printf("viper.Get: %+v\n", viper.Get("resources"))
  println("########################################################################")
}

Here is the output:

Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true
 go build
 ./main
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true
 go build
 ./main
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true
 go build
 ./main
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string][]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string][]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true}}}
Defaults:
map[string]interface {}{}
########################################################################
res1 Name: Super bazzz ; XYZ: true
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"override name"}}}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string][]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true}}}
Defaults:
map[string]interface {}{}
########################################################################
res2 Name: override name ; XYZ: false
########################################################################
########################################################################
res3 Name: override name ; XYZ: false
########################################################################
viper.Get: map[baz:map[name:override name]]
########################################################################

Comment From: missing1984

running into the same issue. any workaround?

Comment From: SVilgelm

there is no workaround, the issue is that the viper has several internal storages. In order to fix the issue, the logic of Get should be changed to merge all possible combinations, like take the baz from the loaded config, then take the baz.name from the overrides and apply it and etc...

Comment From: kasparjarek

Hi, since the release v1.18.0 the issue is also happening for the Unmarshal() function. I suspect that the issue was introduced by this pull request https://github.com/spf13/viper/pull/1429. FYI @krakowski @sagikazarmark

Comment From: hieuvubk

any update on this?

Comment From: chris-dot-exe

Unfortunately we run into this issue too. So I also would be interested on an update to this issue? Any solution in sight?

At the moment the only 'workaround' is to create a completely new viper instance and re-read the config after saving data which obviously isn't a good workaround nor a solution...

Comment From: github-actions[bot]

Issues with no activity for 30 days are marked stale and subject to being closed.

Comment From: SVilgelm

Hi @sagikazarmark, could you please take a look at this issue? I see that there is a fix, I'm not sure if it actually works, but could you please take a look at the PR as well? https://github.com/spf13/viper/pull/1504