Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(util/gconv): add cache logic to enhance performance #3673

Merged
merged 28 commits into from
Sep 9, 2024

Conversation

wln32
Copy link
Member

@wln32 wln32 commented Jul 1, 2024

  1. gconv.Scan, gconv.Struct之类的函数(只要是转换到struct类型的)增加了缓存逻辑,不需要重复解析
  2. 对于常见类型的字段的转换使用gconv.Int, gconv.String之类的,无需多次判断
  3. 缓存特性由UseConvCacheExperiment函数来控制,默认开启,如果开启后有什么bug,关闭即可,平稳过渡
  4. 对于上次 perf(util/gconv): improve performance for struct converting #3412 的改进来说,性能方面大幅度提升,运行速度差不多有4倍,内存方面大幅度减少
  5. 对于需要模糊匹配的情况,增加一个字段来记上次找到的key,大概率可以不再需要模糊匹配
  6. 针对paramsMap的长度选择合适的方法来做赋值
    1. 如果paramsMap的长度比结构体的字段数量少,那么以paramsMap来做循环
    2. 反之用结构体字段map来做循环
      这么所的原因是当paramsMap的长度和结构体字段数量差距过大时,会造成一些不必要浪费
      比如 以下代码
type structType2 struct {
	Name  string
	Score int
	Age   int
	ID    int

	Name1  string
	Score1 int
	Age1   int
	ID1    int
}
m := map[string]any{
	"Name":  "qiang",
	"Score": 600,
	"Age":   98,
	"ID":    199,
}

如果采用结构体字段map的长度来做循环,那么至少需要循环len(结构体字段map)次,即时使用一个额外的字段来统计赋值次数,在合适的时机退出,也不能保证赋值4次就可以退出,因为map的无序性,可能前三个字段一开始就匹配好了,最后一个ID字段可能在第七次才匹配到,中间几次没有匹配到,依然需要模糊匹配,所以这里以paramsMap的长度来做循环比较好
另外一种情况也是同样的


以下是本次的性能,会有一些浮动,

最新提交的性能是
Benchmark_doStruct_Fields8_Basic_MapToStruct
Benchmark_doStruct_Fields8_Basic_MapToStruct-8           3223788              1882 ns/op               9 B/op          0 allocs/op


commit [26c14b3] 之前的
Benchmark_doStruct_Fields8_Basic_MapToStruct
Benchmark_doStruct_Fields8_Basic_MapToStruct-8            976039              6240 ns/op            1310 B/op         10 allocs/op

---
之前pr#3412的
Benchmark_Struct_my_Fields8_Basic_MapToStruct       
Benchmark_Struct_my_Fields8_Basic_MapToStruct-8          482611             11708 ns/op            2074 B/op         44 allocs/op

Benchmark_Struct_gf_Fields8_Basic_MapToStruct                                                                                     
Benchmark_Struct_gf_Fields8_Basic_MapToStruct-8           214789             27090 ns/op            7668 B/op        138 allocs/op

@gqcn gqcn added the awesome It's awesome! We keep watching. label Jul 1, 2024
wln32 and others added 9 commits July 2, 2024 08:07
2.修改或新增一些注释
3.删除无用代码
2.对doStruct的params参数增加map[string]any的快速判断,减少内存申请次数
2.模糊匹配时增加一个字段来记录最后一次匹配到的key,加快匹配
2.增加bool和[]byte两个常见类型的转换函数
@wln32 wln32 changed the title util/gconv: Add gconv.Struct cache logic perf(util/gconv): Add gconv.Struct cache logic Jul 5, 2024
@gqcn
Copy link
Member

gqcn commented Jul 17, 2024

@wln32 很棒的性能优化思路 👍 我刚粗略看了下,整体实现逻辑上没什么问题的,我需要仔细看看学习一下细节,需要花一点时间哈。

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


@wln32 Great performance optimization idea 👍 I just took a quick look at it, and there is nothing wrong with the overall implementation logic. I need to take a closer look at the details and spend some time.

util/gconv/gconv_struct.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct_cache.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct_cache.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct_cache.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct_cache.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct_cache.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct_cache.go Outdated Show resolved Hide resolved
util/gconv/gconv_struct.go Outdated Show resolved Hide resolved
@gqcn
Copy link
Member

gqcn commented Aug 8, 2024

I'm still reviewing this. ❤️

gqcn and others added 2 commits August 14, 2024 21:16
…truct converting (#3)

* fix(cmd/gf): fix command `gf up` with `-u` option upgrading packages indirectly required would fail with higher version of go required (gogf#3687)

* up

* up

* refactor(util/gconv): add struct&field cache to improve performance for struct converting

---------

Co-authored-by: 海亮 <[email protected]>
Copy link

sonarcloud bot commented Sep 9, 2024

@gqcn gqcn changed the title perf(util/gconv): Add gconv.Struct cache logic perf(util/gconv): add cache logic to enhance performance Sep 9, 2024
@gqcn gqcn merged commit 1b4ebc0 into gogf:master Sep 9, 2024
24 checks passed
houseme added a commit that referenced this pull request Sep 12, 2024
…hub.com:gogf/gf into feature/action-1.23

* 'feature/action-1.23' of github.com:gogf/gf:
  feat: version v2.7.3 (#3763)
  perf(util/gconv):  add cache logic to enhance performance (#3673)
  fix(contrib/drivers/pgsql): fix insert error when data struct field has nil in PgSQL (#3679)
  fix(util/gconv): #3731 map type name mismatch in switch case statement (#3732)
  fix(contrib/drivers/pgsql): #3671 fix invalid pgsql insert json type (#3742)
  refactor(nacos-registry): use official nacos sdk instead of the third-party nacos sdk (#3745)

* 'feature/action-1.23' of github.com:gogf/gf:
  feat: version v2.7.3 (#3763)
  perf(util/gconv):  add cache logic to enhance performance (#3673)
  fix(contrib/drivers/pgsql): fix insert error when data struct field has nil in PgSQL (#3679)
  fix(util/gconv): #3731 map type name mismatch in switch case statement (#3732)
  fix(contrib/drivers/pgsql): #3671 fix invalid pgsql insert json type (#3742)
  refactor(nacos-registry): use official nacos sdk instead of the third-party nacos sdk (#3745)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awesome It's awesome! We keep watching.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants