github/golang.net
2
0
Fork 0
mirror of https://github.com/golang/net.git synced 2026-03-31 18:37:08 +09:00
Code Issues Packages Projects Releases Wiki Activity

[release-branch.go1.8] http2: fix lock contention slowdown due to gracefulShutdownCh

Browse Source 
gracefulShutdownCh is shared by all connections in a server. When a
server accumulates many connections (e.g., 5000 in the kubemark-5000
benchmark), we have 5000 serverConn.serve goroutines selecting on this
channel. This means 5000 goroutines hammer the channel's lock, which
causes severe lock contention.

The fix in this CL is to make a local proxy for gracefulShutdownCh in
each connection so that each connection selects on gracefulShutdownCh
at most once per connection rather than once per serverConn.serve loop
iteration.

This fix is intended to be backported quickly into Go 1.8.2. The
downside of this fix is 2KB extra stack usage per connection. A better
fix will be implemented in Go 1.9.

Unfortunately, I have been unable to reproduce this problem locally.
This fix was verified by the kubernetes team. See:
https://github.com/kubernetes/kubernetes/issues/45216#issuecomment-300830243

Updates golang/go#20302

Change-Id: I19ab19268a6ccab9b6e9dffa0cfbc89b8c7d0f19
Reviewed-on: https://go-review.googlesource.com/43455
Run-TryBot: Tom Bergan <tombergan@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
(cherry picked from commit d3ede01617fc9ddf0827da5a431705cf91cf563d)
Reviewed-on: https://go-review.googlesource.com/43459
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Chris Broadfoot <cbro@golang.org>
This commit is contained in:
Tom Bergan
2017-05-12 14:27:43 -07:00
committed by Brad Fitzpatrick
parent 242b6b3517
commit 186fd3fc81
1 changed files with 14 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
http2/server.go
View File

@@ -722,9 +722,13 @@ func (sc *serverConn) serve() {
sc.idleTimerCh = sc.idleTimer.C
}
var gracefulShutdownCh <-chan struct{}
var gracefulShutdownCh chan struct{}
if sc.hs != nil {
gracefulShutdownCh = h1ServerShutdownChan(sc.hs)
ch := h1ServerShutdownChan(sc.hs)
if ch != nil {
gracefulShutdownCh = make(chan struct{})
go sc.awaitGracefulShutdown(ch, gracefulShutdownCh)
}
}
go sc.readFrames() // closed by defer sc.conn.Close above
@@ -773,6 +777,14 @@ func (sc *serverConn) serve() {
}
}
func (sc *serverConn) awaitGracefulShutdown(sharedCh <-chan struct{}, privateCh chan struct{}) {
select {
case <-sc.doneServing:
case <-sharedCh:
close(privateCh)
}
}
// readPreface reads the ClientPreface greeting from the peer
// or returns an error on timeout or an invalid greeting.
func (sc *serverConn) readPreface() error {