创作人 Leo
编辑时间 Wed Nov 23,2022 at 15:03
pprof 图形显示需要 graphviz 程序,如果没有,需要提前安装
https://graphviz.org/download/
调用 go tool pprof -alloc_space {pprof接口地址} 连接进程查看分配情况
输入top来查看累积分配内存较多的一些函数调用
-cum 参数通过使用量排序
$ go tool pprof -alloc_space http://127.0.0.1:8030/debug/pprof/heap
Fetching profile over HTTP from http://127.0.0.1:8030/debug/pprof/heap
Saved profile in /home/dev/pprof/pprof.server.alloc_objects.alloc_space.inuse_objects.inuse_space.007.pb.gz
File: server
Build ID: c870f51b1a90c46b728c406dc80949ef63a535bb
Type: alloc_space
Time: Aug 26, 2020 at 11:27am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top20 -cum
Showing nodes accounting for 122667.15MB, 41.62% of 294697MB total
Dropped 426 nodes (cum <= 1473.49MB)
Showing top 20 nodes out of 50
flat flat% sum% cum cum%
0 0% 0% 183798.25MB 62.37% github.com/apache/rocketmq-client-go/core._cgoexpwrap_b6e997bf7502_consumeMessageCallback
0 0% 0% 183798.25MB 62.37% github.com/apache/rocketmq-client-go/core.consumeMessageCallback
0 0% 0% 183798.25MB 62.37% runtime.cgocallback_gofunc
0 0% 0% 183798.25MB 62.37% runtime.cgocallbackg
0 0% 0% 183798.25MB 62.37% runtime.cgocallbackg1
7147.20MB 2.43% 2.43% 164262.01MB 55.74% encoding/json.Unmarshal
1919.03MB 0.65% 3.08% 159658.34MB 54.18% server/crond.(*TrustMsgFunc).ConsumeWithPush.func1
17286.89MB 5.87% 8.94% 157739.31MB 53.53% server/crond.(*TrustMsgFunc).process
21080.63MB 7.15% 16.10% 155150.76MB 52.65% encoding/json.(*decodeState).object
0 0% 16.10% 155150.76MB 52.65% encoding/json.(*decodeState).unmarshal
0 0% 16.10% 155150.76MB 52.65% encoding/json.(*decodeState).value
0 0% 16.10% 127156.45MB 43.15% encoding/json.(*decodeState).array
1MB 0.00034% 16.10% 107944.81MB 36.63% server/tools/timer.startTicker
0 0% 16.10% 104636.75MB 35.51% server/crond.(*BackendSync).syncCoinPairToRedis
10.50MB 0.0036% 16.10% 104636.75MB 35.51% server/models.GetCoinPair
0 0% 16.10% 78489.54MB 26.63% io/ioutil.ReadAll
0 0% 16.10% 78489.54MB 26.63% io/ioutil.readAll
0 0% 16.10% 78381.49MB 26.60% bytes.(*Buffer).ReadFrom
5.50MB 0.0019% 16.10% 75221.90MB 25.53% bytes.(*Buffer).grow
75216.40MB 25.52% 41.62% 75216.40MB 25.52% bytes.makeSlice
(pprof)
alloc_space 用来查看全部历史内存分配统计,可以通过这个指标确定大量内存用在哪些地方
比如 json 解析,或者数据库操作,然后针对性优化,实例中我们看到主要是 MQ 消息回调使用了大量内存
alloc_inuse 用来查看正在使用的内存,如果发现有代码内存过大,则可能存在内存泄漏问题,GC没有及时回收应该回收的内存
$ go tool pprof -inuse_space http://127.0.0.1:8030/debug/pprof/heap
...
(pprof) top -cum
Showing nodes accounting for 517.47MB, 98.10% of 527.48MB total
Dropped 56 nodes (cum <= 2.64MB)
flat flat% sum% cum cum%
0 0% 0% 519.97MB 98.58% server/tools/timer.startTicker
376.02MB 71.29% 71.29% 519.47MB 98.48% server/crond.(*NoticeMsgFunc).Worker
0 0% 71.29% 141.45MB 26.82% server/netws.PublishMSG
0 0% 71.29% 141.45MB 26.82% server/netws.eventHandleCellNet
141.45MB 26.82% 98.10% 141.45MB 26.82% github.com/davyxu/cellnet.(*Pipe).Add
0 0% 98.10% 141.45MB 26.82% github.com/davyxu/cellnet/peer/gorillaws.(*wsSession).Send
0 0% 98.10% 5.51MB 1.04% net/http.(*conn).serve
0 0% 98.10% 3.01MB 0.57% net/http.(*ServeMux).ServeHTTP
0 0% 98.10% 3.01MB 0.57% net/http.HandlerFunc.ServeHTTP
0 0% 98.10% 3.01MB 0.57% net/http.serverHandler.ServeHTTP
这段代码是最近处理的一个内存溢出问题的pprof快照
可以看到 Worker 程序存在内存溢出出问题
有很多种情况会导致内存溢出问题,这里列举几个:
go pprof 还有一些参数能够生成火焰图和调用流程图,在生产环境使用的话可能需要运维配合开端口
知道目前有多少正在运行的协程很重要,查看是否协程数量异常,是排查内存溢出问题的一种实用方法
因为大部分时候如果资源句柄没关闭,或者缓冲区没及时释放,只要关闭使用的协程,这些指针引用就不存在了,自然会被GC回收掉
$ go tool pprof http://127.0.0.1:8030/debug/pprof/goroutine
Fetching profile over HTTP from http://127.0.0.1:8030/debug/pprof/goroutine
Saved profile in /home/dev/pprof/pprof.server.goroutine.003.pb.gz
File: server
Build ID: c870f51b1a90c46b728c406dc80949ef63a535bb
Type: goroutine
Time: Aug 18, 2020 at 11:21am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top20 -cum
Showing nodes accounting for 766, 99.74% of 768 total
Dropped 39 nodes (cum <= 3)
Showing top 20 nodes out of 33
flat flat% sum% cum cum%
766 99.74% 99.74% 766 99.74% runtime.gopark
0 0% 99.74% 374 48.70% internal/poll.(*pollDesc).wait
0 0% 99.74% 374 48.70% internal/poll.(*pollDesc).waitRead
0 0% 99.74% 374 48.70% internal/poll.runtime_pollWait
0 0% 99.74% 374 48.70% runtime.netpollblock
0 0% 99.74% 372 48.44% runtime.chanrecv
0 0% 99.74% 372 48.44% runtime.goparkunlock
0 0% 99.74% 371 48.31% internal/poll.(*FD).Read
0 0% 99.74% 371 48.31% net.(*conn).Read
0 0% 99.74% 371 48.31% net.(*netFD).Read
0 0% 99.74% 370 48.18% bufio.(*Reader).fill
0 0% 99.74% 369 48.05% bufio.(*Reader).Peek
0 0% 99.74% 369 48.05% runtime.chanrecv2
0 0% 99.74% 368 47.92% net/http.(*conn).serve
0 0% 99.74% 368 47.92% net/http.serverHandler.ServeHTTP
0 0% 99.74% 367 47.79% server/netws.defaultWSLoopConnectWS
0 0% 99.74% 367 47.79% server/netws.startWriteListener
0 0% 99.74% 367 47.79% github.com/gorilla/websocket.(*Conn).NextReader
0 0% 99.74% 367 47.79% github.com/gorilla/websocket.(*Conn).ReadMessage
0 0% 99.74% 367 47.79% github.com/gorilla/websocket.(*Conn).advanceFrame
(pprof)
解读: 这里需要注意,不管是协程还是内存分析,cum 的数量都是包含被调用函数使用内存/协程的累计值 比如 net/http.serverHandler.ServeHTTP 是 368 协程,其中包含了 server/netws.defaultWSLoopConnectWS 的 367 协程
生成 profile 文件
func main() {
f, _ := os.OpenFile("cpu.pprof", os.O_CREATE|os.O_RDWR, 0644)
defer f.Close()
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
// ....
}
分析数据
$ go tool pprof -http=:9999 cpu.pprof
Serving web UI on http://localhost:9999
参考
golang 内存分析/动态追踪
通过 profiling 定位 golang 性能问题 - 内存篇
实战go内存泄漏
High Performance Go Workshop
pprof 性能分析