Introducing Go-MMProxy: a Layer-3 Solution
In this latest blog post, we will be exploring the work that has been done by Path’s talented and dedicated engineers. The product of much labor and time, we would like to introduce go-mmproxy. This advancement is a Go-based proxy tool, created to improve upon the proxy solutions currently available and offer actual run-time stability while providing top-tier performance in terms of connection and packet throughput. (1)
To add some context, we undertook this project for the purpose of creating a better and more efficient proxy tool for clients to run on their servers in order to get connections coming from real end-user IP addresses. Otherwise, only the intermediate server addresses would show up. When utilizing the current solutions available, however, many have had connectivity issues and also issues with crashing. Knowing this, we set out to make a better proxy tool for our uses and tailor it to fit better for our needs and clients. Our results in this effort surprised even our engineers.
Above, you can see the benchmark results that our team assembled to show exactly how much of an improvement our go-mmproxy is compared to Cloudflare's mmproxy, which is a noted proxy tool. We display a large difference in Mbps rate and packet throughput between the two in the data above. The issues discussed by many others regarding mmproxy, along with the actual performance issues that are evident in the data above, show that it is simply an untenable solution for anyone that is trying to truly optimize.
To provide more of a background, go-mmproxy is a standalone application that unwraps HAProxy's PROXY protocol so that the network connection to the end server comes from the IP address and port number of the client, instead of the proxy server. Furthermore, it is a PROXY protocol gateway, as go-mmproxy listens for remote connections coming from an application level load balancer. It then reads a PROXY protocol header, opens a localhost connection to the target application, and duly proxies data in and out. Such a proxy wouldn't be too useful if not for one feature—the localhost connection from go-mmproxy to the target application is sent with a real client source IP.
What this means is, essentially, go-mmproxy spoofs the client IP address. From the application’s point of view, this spoofed connection, is indistinguishable from a real one, connecting directly to the application.
Path’s client-facing purpose in the creation of this proxy tool is for use on the Shotbow Minecraft server. This was an area of initial interest due to video game servers being a major, obvious use case for go-mmproxy. Primarily due to this being a sector in which you geniunely want to see real client’s IP’s, if only to be able to ban them. In the beginning, after much toying and testing however, we discovered mmproxy to be a severely lacking implementation. Cases abound of it crashing for the customer, resulting in all of the connections being dropped and the customer being unhappy, among other deterrent results. It is notable, however, that others have experienced similar problems, as mmproxy crashing is a well-known issue. (3)
Aimed at creating our own proxy tool that would be free of the issues above, Path’s engineers focused intently and set out to create a customized implementation to improve on mmproxy’s run-time stability while providing potentially greater performance in terms of connection and packet throughput. Go (language) seemed to be a natural fit at that point, primarily because mmproxy is a relatively simple application that can potentially be asked to proxy hundreds of thousands concurrent connections, which is not unlike a "hello world" app in Go. In fact, Cloudflare’s mmproxy uses "libmill" library for C language, which attempts to provide some core Go concurrency features in C; notably, our engineers suspect it is the quality of this library which leads to the myriad crashes.
Overall, the stability of go-mmproxy is hard to truly measure due to sample size, but we think it's sufficient to say that it has been running in front of Shotbow servers for over a month without a single issue. While being satisfied with this stability progress, writing our own implementation allowed the team to go a bit further than just stability improvements. First of all, code efficiency was paramount during development. Second, go-mmproxy now supports both PROXY protocol (v1) and the newer and more efficient PROXY protocol (v2). Third, we now include a *beta* support for UDP. This is also related to the second point due to the fact that PROXY protocol (v1) only supports TCP connections. Overall, the points outlined above show the importance and utility of this work, as well as the benefits that it offered for our uses. We hope to see even more advancements from the community in this space in the near future.