go pprof 工具

pprof 图形显示需要 graphviz 程序，如果没有，需要提前安装
https://graphviz.org/download/

内存

调用 go tool pprof -alloc_space {pprof接口地址} 连接进程查看分配情况
输入top来查看累积分配内存较多的一些函数调用
-cum 参数通过使用量排序

$ go tool pprof -alloc_space http://127.0.0.1:8030/debug/pprof/heap
Fetching profile over HTTP from http://127.0.0.1:8030/debug/pprof/heap
Saved profile in /home/dev/pprof/pprof.server.alloc_objects.alloc_space.inuse_objects.inuse_space.007.pb.gz
File: server
Build ID: c870f51b1a90c46b728c406dc80949ef63a535bb
Type: alloc_space
Time: Aug 26, 2020 at 11:27am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top20 -cum
Showing nodes accounting for 122667.15MB, 41.62% of 294697MB total
Dropped 426 nodes (cum <= 1473.49MB)
Showing top 20 nodes out of 50
      flat  flat%   sum%        cum   cum%
         0     0%     0% 183798.25MB 62.37%  github.com/apache/rocketmq-client-go/core._cgoexpwrap_b6e997bf7502_consumeMessageCallback
         0     0%     0% 183798.25MB 62.37%  github.com/apache/rocketmq-client-go/core.consumeMessageCallback
         0     0%     0% 183798.25MB 62.37%  runtime.cgocallback_gofunc
         0     0%     0% 183798.25MB 62.37%  runtime.cgocallbackg
         0     0%     0% 183798.25MB 62.37%  runtime.cgocallbackg1
 7147.20MB  2.43%  2.43% 164262.01MB 55.74%  encoding/json.Unmarshal
 1919.03MB  0.65%  3.08% 159658.34MB 54.18%  server/crond.(*TrustMsgFunc).ConsumeWithPush.func1
17286.89MB  5.87%  8.94% 157739.31MB 53.53%  server/crond.(*TrustMsgFunc).process
21080.63MB  7.15% 16.10% 155150.76MB 52.65%  encoding/json.(*decodeState).object
         0     0% 16.10% 155150.76MB 52.65%  encoding/json.(*decodeState).unmarshal
         0     0% 16.10% 155150.76MB 52.65%  encoding/json.(*decodeState).value
         0     0% 16.10% 127156.45MB 43.15%  encoding/json.(*decodeState).array
       1MB 0.00034% 16.10% 107944.81MB 36.63%  server/tools/timer.startTicker
         0     0% 16.10% 104636.75MB 35.51%  server/crond.(*BackendSync).syncCoinPairToRedis
   10.50MB 0.0036% 16.10% 104636.75MB 35.51%  server/models.GetCoinPair
         0     0% 16.10% 78489.54MB 26.63%  io/ioutil.ReadAll
         0     0% 16.10% 78489.54MB 26.63%  io/ioutil.readAll
         0     0% 16.10% 78381.49MB 26.60%  bytes.(*Buffer).ReadFrom
    5.50MB 0.0019% 16.10% 75221.90MB 25.53%  bytes.(*Buffer).grow
75216.40MB 25.52% 41.62% 75216.40MB 25.52%  bytes.makeSlice
(pprof) 

alloc_space 用来查看全部历史内存分配统计，可以通过这个指标确定大量内存用在哪些地方
比如 json 解析，或者数据库操作，然后针对性优化，实例中我们看到主要是 MQ 消息回调使用了大量内存
alloc_inuse 用来查看正在使用的内存，如果发现有代码内存过大，则可能存在内存泄漏问题，GC没有及时回收应该回收的内存

$ go tool pprof -inuse_space http://127.0.0.1:8030/debug/pprof/heap
...
(pprof) top -cum
Showing nodes accounting for 517.47MB, 98.10% of 527.48MB total
Dropped 56 nodes (cum <= 2.64MB)
      flat  flat%   sum%        cum   cum%
         0     0%     0%   519.97MB 98.58%  server/tools/timer.startTicker
  376.02MB 71.29% 71.29%   519.47MB 98.48%  server/crond.(*NoticeMsgFunc).Worker
         0     0% 71.29%   141.45MB 26.82%  server/netws.PublishMSG
         0     0% 71.29%   141.45MB 26.82%  server/netws.eventHandleCellNet
  141.45MB 26.82% 98.10%   141.45MB 26.82%  github.com/davyxu/cellnet.(*Pipe).Add
         0     0% 98.10%   141.45MB 26.82%  github.com/davyxu/cellnet/peer/gorillaws.(*wsSession).Send
         0     0% 98.10%     5.51MB  1.04%  net/http.(*conn).serve
         0     0% 98.10%     3.01MB  0.57%  net/http.(*ServeMux).ServeHTTP
         0     0% 98.10%     3.01MB  0.57%  net/http.HandlerFunc.ServeHTTP
         0     0% 98.10%     3.01MB  0.57%  net/http.serverHandler.ServeHTTP

这段代码是最近处理的一个内存溢出问题的pprof快照
可以看到 Worker 程序存在内存溢出出问题
有很多种情况会导致内存溢出问题，这里列举几个：

1. 未关闭资源句柄，比如 orm 返回的数据读句柄、http Response.body 句柄
2. 协程未及时退出，导致协程中的数据一直存在，这也是比较常见的，所以大家都在提倡 ‘知道协程会在什么时候退出’
   
3. 由于GO垃圾回收基于引用计数和三色标记多阶段回收，所以如果还有协程持有该内存地址，就不会被回收

go pprof 还有一些参数能够生成火焰图和调用流程图，在生产环境使用的话可能需要运维配合开端口

协程

知道目前有多少正在运行的协程很重要，查看是否协程数量异常，是排查内存溢出问题的一种实用方法
因为大部分时候如果资源句柄没关闭，或者缓冲区没及时释放，只要关闭使用的协程，这些指针引用就不存在了，自然会被GC回收掉

$ go tool pprof http://127.0.0.1:8030/debug/pprof/goroutine
Fetching profile over HTTP from http://127.0.0.1:8030/debug/pprof/goroutine
Saved profile in /home/dev/pprof/pprof.server.goroutine.003.pb.gz
File: server
Build ID: c870f51b1a90c46b728c406dc80949ef63a535bb
Type: goroutine
Time: Aug 18, 2020 at 11:21am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top20 -cum
Showing nodes accounting for 766, 99.74% of 768 total
Dropped 39 nodes (cum <= 3)
Showing top 20 nodes out of 33
      flat  flat%   sum%        cum   cum%
       766 99.74% 99.74%        766 99.74%  runtime.gopark
         0     0% 99.74%        374 48.70%  internal/poll.(*pollDesc).wait
         0     0% 99.74%        374 48.70%  internal/poll.(*pollDesc).waitRead
         0     0% 99.74%        374 48.70%  internal/poll.runtime_pollWait
         0     0% 99.74%        374 48.70%  runtime.netpollblock
         0     0% 99.74%        372 48.44%  runtime.chanrecv
         0     0% 99.74%        372 48.44%  runtime.goparkunlock
         0     0% 99.74%        371 48.31%  internal/poll.(*FD).Read
         0     0% 99.74%        371 48.31%  net.(*conn).Read
         0     0% 99.74%        371 48.31%  net.(*netFD).Read
         0     0% 99.74%        370 48.18%  bufio.(*Reader).fill
         0     0% 99.74%        369 48.05%  bufio.(*Reader).Peek
         0     0% 99.74%        369 48.05%  runtime.chanrecv2
         0     0% 99.74%        368 47.92%  net/http.(*conn).serve
         0     0% 99.74%        368 47.92%  net/http.serverHandler.ServeHTTP
         0     0% 99.74%        367 47.79%  server/netws.defaultWSLoopConnectWS
         0     0% 99.74%        367 47.79%  server/netws.startWriteListener
         0     0% 99.74%        367 47.79%  github.com/gorilla/websocket.(*Conn).NextReader
         0     0% 99.74%        367 47.79%  github.com/gorilla/websocket.(*Conn).ReadMessage
         0     0% 99.74%        367 47.79%  github.com/gorilla/websocket.(*Conn).advanceFrame
(pprof)

解读： 这里需要注意，不管是协程还是内存分析，cum 的数量都是包含被调用函数使用内存/协程的累计值 比如 net/http.serverHandler.ServeHTTP 是 368 协程，其中包含了 server/netws.defaultWSLoopConnectWS 的 367 协程

CPU

生成 profile 文件

func main() {
	f, _ := os.OpenFile("cpu.pprof", os.O_CREATE|os.O_RDWR, 0644)
	defer f.Close()
	pprof.StartCPUProfile(f)
	defer pprof.StopCPUProfile()

      // ....
}

分析数据

$ go tool pprof -http=:9999 cpu.pprof
Serving web UI on http://localhost:9999

参考

golang 内存分析/动态追踪
通过 profiling 定位 golang 性能问题 - 内存篇
实战go内存泄漏
High Performance Go Workshop
pprof 性能分析