More powerful load balancers
You should improve your load balancer to support higher session rates (especially with SSL). We would be happy to pay extra money (e.g. $40, $80, $160, etc.) to have load balancers with high performance.
Currently this is a huge problem for us: we cannot scale.
At the moment your load balancers are a bottleneck and cannot handle more than 1,000 requests / s (without keep alive and with SSL enabled).
The same limit of 1,000 requests / s also applies to custom installations of HAProxy on DO... probably your hardware is not optimized for SSL and scaling the droplet / processes doesn't solve the issue.
Thanks for your feedback! We are aware of some limitations and constraints on our current offering.
We are currently investigating and planning to implement better scalability and performance for our Load Balancer service in the coming months.
Marco Colli commented
In any case, after some weeks of investigation, I am now pretty sure that the limitations come from your hardware performance. I see that my results are near to the hardware limits for ssl termination:
So I think that the only thing that can help are better CPUs. I don't know if providing LB with more virtual CPUs scales well.
I think that if you just increase the number of virtual CPUs (and not the quality of hardware) you should test SSL termination to make sure that performance (new connections / s, handshakes / s) are actually improved (the answer from my tests is "no").
Marco Colli commented
@Tyler Crandall Thanks for the quick reply!
In production we currently handle a session rate of 600 new SSL connections / s. Based on load tests, we cannot handle a session rate higher than 1,000 new SSL connections / s.
Even with all the HAProxy optimizations* we cannot get session rates higher than 1,000.
We have tried to increase the number of CPUs (tested with 12 CPUs) and the number of HAProxy processes accordingly, but the results are the same: we cannot handle session rates higher than 1,000 new SSL connections / s.
The same limits (or even lower) apply to your own load balancer that can be created from the dashboard.
The limit is not in the backend because it was tested with much more requests: we have tested a single backend server successfully with 7,000 requests / s with a static file served by nginx on HTTP. Note that we have 10 backend servers, so, for a static file, we should be able to serve 70,000 requests / s.
Actually we don't need a session rate of 70,000 requests / s for the frontend... because most requests are not simple static files, so the real throughput of the backend is lower. However we will soon need to handle several thousands of SSL connections / s (e.g. session rate = 5,000) and that is not currently possible on DigitalOcean.
I would be happy to provide additional details: feel free to contact me directly by email. Currently I have received a message from you, but I haven't found a way to reply, so I reply here.