Attention All Hands,
The last few weeks we have been testing and changing many settings and possibilities with our servers, which have not been working in the year of 2018 as we expect them too. These can be drawn back to a multitude of reasons, most especially mod and ArmA updates, but that doesn't really tell the whole story. I will attempt to break that down somewhat, as best I can.
There is really not much we can do here. We are all at the mercy of the updates to ArmA, and any changes they do are effective to everyone immediately, and there is no stopping it. In ArmA 1 and ArmA 2 we were able to tell people to hold off of updates until those updates had been vetted, and any possible hotfixes had been made, if needed. Unfortunately the nature of ArmA 3 and Steam makes this impossible now. As such we are at their mercy. We have recently switched to Steam distribution of our major mods (CUP Terrains, ACE, RHS) in order to keep us in line with these major updates. If a major update is applied to ArmA, we are able to follow the immediate update of these major mods through their Steam distribution, without worry of missing them because it's too late to update our repo or to get people to update.
Unfortunately there is nothing much more we can do on this front. We can be more diligent on the types of mods we add to our pack (ie. the SU-34 that was added to the OPFOR pack and was explicitly told not to be used... but why was it added in the first place in that case? Or the invisible MiGs) but the updating and any possible instability from those updates are something we cannot do anything about until it happens, really (like the cTab memory leak).
Many people have brought up concerns of hardware. Allow me to give you a breakdown of our server hardware:
1 Gbps Dedicated Port Bandwidth Unlimited (10TB Full Speed) 16 GB DDR4 RAM Intel Core i7 6700K 256 GB Primary SSD Drive Windows Server 2012 R2 Standard DDoS Protection 20 Gbps DDoS Protection
This is a very powerful server, more than suitable for ours, or anyones needs. This is not a question of hardware.
However. We have, regardless, purchased a secondary server for testing purposes, on the hypothesis that there is an issue with the data center. This server is what Task Force Bravo was operating on during OP 02-18. While the server performed well, it was inconclusive as to whether or not this new server is the/an answer. That will not be clear until after OP 03 or OP 04, depending on the outcomes of tests during those operations.
We have talked a lot about server settings during this fiasco. A lot of care and attention has been paid to these settings, being tweaked here and there to attempt to get that sweet spot. I could get in to what we've done with 'Max Guaranteed Bandwidth' and 'Minimum Messages' and a whole bunch of crap.. but I won't. Mostly because it's boring and I don't even understand it all personally. Suffice to say in the majority of settings we have what is required and needed, but we need to talk about one specific setting: Max Ping and Desync.
I spoke about tests earlier. Many of you know that for the past few operations we have been using the MSOT server box to load balance our servers in attempt to see that the issues are. In OP 01-18 Task Force Alpha operated on the Unit Server Box, while Task Force Bravo operated on the MSOT Server Box. Both servers ran the exact same settings, with the exception of this ping/desync kick setting. TFA did not have this enabled. TFB had this enabled for all pings over 450, and desync for over 9000. TFB ran at 45 FPS all night, with no issues. TFA crashed. Catastrophically.
This is when Maj Thomas purchased a new server box (hereafter referred to as the Thomas Server Box). OP 02-18 saw TFA operating on the MSOT Server Box, and TFB operating on the Thomas Server Box. Both servers had the exact same settings, including the ping/desync kicker. Both servers ran at 45 FPS all night. However because of the lack of OPFOR, this is an inconclusive test, though one that does draw some very concerning conclusions.
OP 03-18 is going to see a follow-up tests. Company HQ and 1stLt McNeely are already aware of this impending test, as well as anyone who read my post in the Tun Tavern (forums or Discord). OP 03-18 will again see TFA on the MSOT Server Box, running the same settings. TFB will be operating on the Thomas Server Box, however we will be removing the ping/desync kicker. There are two possibilities of the outcome:
1) TFB runs with no issues. This points to either an issue with the Unit Server Box or the specific personnel makeup of Task Force Alpha, with a handful of individuals causing the lacking server performance
2) TFB runs with many issues. This points to a fundamental change in the netcode of ArmA, no longer being so forgiving to those with higher desync and ping.
In the case of outcome #1, we will in OP 04-18 be placing TFB on the MSOT Server Box with the 'control' settings, and TFA on the Thomas Server Box without the ping/desync kicker. If it performs without issue, then we know the Unit Server Box was an issue, and it will be cancelled, and MEU HQ will foot the bill for the Thomas Server Box and an additional server box to compensate (the Thomas Server Box is a smaller VPS that is less able to run two full-sized Task Forces at once) until September, when donations were collected through. If it performs with issues, then we know it is a specific personnel composition to Task Force Alpha.
In the case of outcome #2, we will be forced to permanently implement the ping/desync kicker on our Task Force Servers. This will lead to many of our Asian, Australian, Eastern European, and some Western European members being unable to join us anymore, and close off that avenue for a long time to come. If it comes to this, we will be discussing the opening of a sister/child unit catering specifically to these regions, or directing those interested to other units with which we already have close contact that cater to these regions. This will be a public discussion for all of those affected.
I understand you are all frustrated. Believe me, so am I. Most of you understand that we have been working on this furiously, however there are some of you who would like to ignore that fact. This Unit has been through many such trials and tribulations that ArmA likes to throw at us. You can cower and point fingers and cry and quit if you like, lacking the fortitude and will to weather issues like your brothers and sisters of the Unit deserve; or you can buckle down and raise one another up. We all come here for Sundays, obviously, and we will always do everything we can to make those enjoyable and memorable experiences. But we all stay here for the ones beside us. Don't forget that.
We are on the cusp of having this fully solved. It may be May, basically June now, but this has been a more... interesting... year that ArmA has given us. So let's make the second half our little bitch, shall we?