[PATCH] reuse_port option for multiple server instances

[Harry Mason]

On Mon, 2016-01-11 at 09:44 -0700, Alex Rousskov wrote:
> On 01/11/2016 07:09 AM, Harry Mason wrote:
> > This patch allows multiple polygraph-server instances to bind to the
> > same address and port.
>
> Thank you! Please note that according to our tests with Squid on
> Linux[1], sharing the listening socket does not automatically balance
> the load among the listening processes due to unfortunate TCP stack
> listener selection algorithms in the kernel.
> [1] http://wiki.squid-cache.org/Features/SmpScale#Will_similar_workers_receive_similar_amount_of_work.3F

I observed that too, but in my particular case it's not very important.

> I believe the last paragraph in the article you cited[2] may hint at
> a similar problem.
> [2] https://lwn.net/Articles/542738/

That seemed to me to be arguing that a multithreaded solution would be
unbalanced, but SO_REUSEPORT ought to be balanced. From the Squid page
it sounds like SO_REUSEPORT is intended to be balanced, and the reason
why it is not is unclear. Apparently the kernel uses a hash of the
tuple to decide which listener should wake up [3]; this explains why
connections may drop when a new listener is started, as that perturbs
the hash, and also that the kernel does not try to compensate if one
listener happens to be busier.
[3] https://lwn.net/Articles/542629/

> Do you see any harm in enabling server port reuse on all platforms
> where it is supported, without adding a PGL option to control this
> behavior?

I don't know which platforms it would work on. For instance it sounds
like FreeBSD has different semantics for SO_REUSEPORT [4]. To me it
sounds sensible to set the option all the time on supported platforms,
but I'm not very familiar with how Polygraph might be deployed.
[4] http://lists.freebsd.org/pipermail/freebsd-net/2013-July/036131.html

> The only negative side effect I can see is where the test is
> misconfigured to listen on a server port already used by another
> program (e.g., httpd). In that case, reusing the port may mask a
> serious configuration flaw, right?

Only if the original bind also set SO_REUSEPORT and shares the same
effective user ID. Otherwise the second bind will fail anyway.

Cheers,
Harry