I want to query database data to the map [string] interface {}. The total amount of data is approximately 170w, with a total data size of 800M. However, during the query process, the memory may increase to 16GB, for unknown reasons.
sql：
var dataMapTmp []map[string]interface{}
err := mysql.Db.Table(m.TableName()).Offset(i).Limit(step).Find(&dataMapTmp).Error
pprof:
Comment From: github-actions[bot]
The issue has been automatically marked as stale as it missing playground pull request link, which is important to help others understand your issue effectively and make sure the issue hasn't been fixed on latest master, checkout https://github.com/go-gorm/playground for details. it will be closed in 30 days if no further activity occurs. if you are asking question, please use the Question template, most likely your question already answered https://github.com/go-gorm/gorm/issues or described in the document https://gorm.io ✨ Search Before Asking ✨
Comment From: github-actions[bot]
The issue has been automatically marked as stale as it missing playground pull request link, which is important to help others understand your issue effectively and make sure the issue hasn't been fixed on latest master, checkout https://github.com/go-gorm/playground for details. it will be closed in 30 days if no further activity occurs. if you are asking question, please use the Question template, most likely your question already answered https://github.com/go-gorm/gorm/issues or described in the document https://gorm.io ✨ Search Before Asking ✨
Comment From: voodoo-dn
@xu-mengnan Hi. What is your version of Gorm?
Today I've experienced OOM Kill by Kubernetes during load on DB, usually it's "read" load to replica. Think maybe this is the same issue.
Because usually container uses ~36mb, but killed after reaching 512mb.
Comment From: github-actions[bot]
The issue has been automatically marked as stale as it missing playground pull request link, which is important to help others understand your issue effectively and make sure the issue hasn't been fixed on latest master, checkout https://github.com/go-gorm/playground for details. it will be closed in 30 days if no further activity occurs. if you are asking question, please use the Question template, most likely your question already answered https://github.com/go-gorm/gorm/issues or described in the document https://gorm.io ✨ Search Before Asking ✨
Comment From: xu-mengnan
@xu-mengnan Hi. What is your version of Gorm?
Today I've experienced OOM Kill by Kubernetes during load on DB, usually it's "read" load to replica. Think maybe this is the same issue.
Because usually container uses ~36mb, but killed after reaching 512mb.
My import version is the following content: - gorm.io/driver/mysql v1.4.7 - gorm.io/gorm v1.24.6
Comment From: github-actions[bot]
The issue has been automatically marked as stale as it missing playground pull request link, which is important to help others understand your issue effectively and make sure the issue hasn't been fixed on latest master, checkout https://github.com/go-gorm/playground for details. it will be closed in 30 days if no further activity occurs. if you are asking question, please use the Question template, most likely your question already answered https://github.com/go-gorm/gorm/issues or described in the document https://gorm.io ✨ Search Before Asking ✨
Comment From: atghw
Had a similar issue with the following:
type Settings struct {
Instance string `json:"instance" gorm:"primaryKey"`
ExtId uint `json:"ext_id" gorm:"primaryKey"`
... somesthing ..
}
var settings []Settings
db := GetDB()
for _, info := range infos {
db = db.Or(
db.Where("ext_id = ?", info.ExtId).Where("instance = ?", info.Instance),
)
}
result := db.Find(&settings) // <--- Problem
Problem: The call to db.Find() didn't returned and memory allocation of the process increased to endless.
I tried to rewrite the query and used db.Rows()with the same condition, but it failed too. I ended up in a loop to fetch them one-by-one. My import versions:
- gorm.io/driver/mysql v1.5.7
- gorm.io/gorm v1.25.11
I cross-compiled on macOS for linux/amd64.