Results 1 to 11 of 11

Thread: What causes servers to lock up/max out/stop functioning

  1. #1
    Veteran Combat pilot
    Join Date
    May 2015
    Location
    Kansas City, Missouri area
    Posts
    460
    Post Thanks / Like
    Total Downloaded
    84.81 MB

    What causes servers to lock up/max out/stop functioning

    Some of you may know that we have had issues over the past couple of years with the TWC servers where the server just gets sort of maxed out and stops functioning. Suddenly it will take 10-15 minutes to enter the server, any communication to the server might be quite slow, and so on.

    The strange thing is, it usually happens when I make some rather small changes that don't seem like they would cause a dramatic effect at all.

    I've been gradually getting my head around what must be the root cause of this, and another incident this weekend has completely confirmed it in my mind.

    So, briefly, here are the conclusions:

    • It's not the number of aircraft per se - neither breathers nor ai nor the combination of the two. It is certainly possible to get overloaded with aircraft but the symptoms are different and the failure is more graceful and gradual than sudden. FYI a powerful server can handle 150-200 aircraft in-game quite well, 200-250 maybe but not optimal, and then 250+ is where things get a lot more dicey. But if you run into the problem I outline below you can remove 50 aircraft with no noticeable effect and say 100 with barely noticeable effect.
    • Ground stationaries in general don't seem to have much effect - particularly if they're "just scenery" that doesn't interact with players, for example by shooting at them. It may take a little longer to get these loaded on entering the server but the penalty for say tens of thousands of statics vs just a couple of hundred is remarkably small. When I've encountered this problem I've tried removing many thousands of ground stationaries with no noticeable effect at all.
    • When you have the problem I'm describing, you can monitor it on the server by simply watching server CPU load. The problem occurs when the server CPU is maxed out. With multi-core CPUs of course you won't see 100% CPU but you'll see one core maxed out with the main server thread and that is when you'll find yourself getting into trouble. The server can do some limited multi-tasking but like if you have a quad-core CPU, you start your server, and you find launcher64.exe is continually taking 25%+ CPU (let's say before any players have even entered) then you're likely in trouble. If you were even at 18% rather than 25-30% you might have enough breathing room but if it's more like 25%+ at "idle" then you're likely in trouble. More to the point, you can monitor things rather well just by keeping an eye on that CPU percentage on the server.

      Just for example, one symptom of this problem is very long server load times for players joining the server. You'll see for example, CPU doesn't look maxed out and players join the server in 2-3 minutes. When the server is maxed out or a bit beyond, suddenly the players take 10-15-20 minutes to join. The "join" bar gets stuck at 99% for a long, long, long time.

      Players complain that "you have too many things in your server". But that is not the root cause at all. You can add 20 or 30K ground stationaries and then remove them without making a noticeable difference. You can add 150 aircraft and remove them and again, no noticeable difference.

      It's not "things in the server" per se but one particular type of thing that causes this particular problem.
    • Related, I noticed there were clear thresholds where a certain version of the server wouldn't run at all on Computer A, would barely run on Computer B, and would run just fine on Computer C. The difference in every case was simply raw CPU power. Computer C had enough CPU cycles to handle this particular type of load (see below), which seems to be fairly steady-state regardless of how many players are in the server etc, with just enough cycles left over to run everything else, too. (All the other thing that go on in the server, like AI aircraft, player aircraft, etc etc, clearly take a far lower percentage of the CPU, or perhaps they scale more gracefully somehow as max load is approached.) The CPUs on computers A & B couldn't even keep up with this load and had nothing left over for the rest, thus what ran just fine on Computer C would barely run or not run at all on the less capable CPUs.
    • The lock-up situation seems to have a rather distinct threshold effect. Like if you keep adding more aircraft to the server you'll find a sort of gradual drop-off of responsiveness. But if you add more of this problem element (see what it is below) you'll have some hundreds to thousands of them in-game and all is well, and you add just a few more (10? 20? maybe 100? whatever it is, a small amount added in proportion to the amount you already have) and suddenly you go from "perfectly functional working server" to "completely disastrous non-responsive server"
    • I have long suspected that AA/artillery is the ultimate cause of this problem. You can test this by gradually adding more & more AA/Artillery to a server and sure enough you find behavior like I outline above - up to a point everything works fine and then you reach a definite threshold and suddenly your server is completely non-functional. You can get to a point where you can't even join the server even though there is nothing going on in it.
    • So AA & artillery is culprit #1.
    • But what I hadn't thought through is a large number of other game elements act just like AA/artillery (and must have the same underlying code to a great degree) and so they add to the problem exactly as through they are AA/artillery. These include:

      • Ships - anything that has a gun or shoots (some kinds of barges & transport ships etc) but even more so, MILITARY ships. Those things are like 10 batteries of artillery. Some of them can fire at a tremendous rate and have a bunch of different guns all on one single platform. So, one military ship counts for maybe a dozen or a few dozen individual AA/artillery - depending on the exact ship and armament.
      • Ground vehicles. Again - anything that fires or has a gun. So for example, if you have a ground convoy of six or eight vehicles. Each vehicle can pack a lot of firepower (depending on which exact vehicle you choose) and some fire at a high rate of speed. So again, half a dozen convoy vehicles might be worth a dozen or two artillery or AA pieces.
      • Finally, trains seem to punch above their weight in terms of CPU load. I don't know that they are exactly in the same class as the artillery/AA and similar shoot-ey things, but they are more problematic than you would think. At one point we had like 3 trains running on each side (so 6 total) and found that you could either have those 6 trains or like 100 AI aircraft - that was about the tradeoff.


    There might be a few more things on the list, but the point is, anything that aims, shoots, and fires is going to use a percentage of the available CPU power. AA/artillery, ships whether stationary or moving (particularly military ships), and convoy type vehicles that shoot and defend themselves - all will add to the problem and at the point you have "too many" of these your server will go from working nicely to broken in about 1 second flat.

    You have a clearly limited amount of "artillery power" available on the server, depending on your server's CPU capability, and if you exceed the workable amount you'll see a rather quick transition from "functions well" to "doesn't function at all".

    At any rate, that's my observation. If you have a server that was running find, you made a few changes and suddenly it's laggy and takes forever for players to log in, look at these "shooter" type elements first.

    Just for example, I added what I thought of as "just a few" ships and suddenly our server went from functional to dysfunctional. (Of course, some of the ships were military, so count them as like 4 dozen AA/artillery pieces.)

    Another time I added "a few" convoys, with similar effect. This was about 4 convoys with 6 or 8 vehicles in them. That seems like a small amount to me. (Also if you added 10 dozen of these items as static "stationaries" you wouldn't see any noticeable change in server functionality. But the live moving/shooting version looks similar but has a far, far different requirement as far as CPU power.) And once again it's not just that these have to be thought of as "aha, another artillery/AA but mobile." On the contrary, these items seem to punch somewhat above their weight as far as CPU usage, when compared with AA or artillery pieces. One of these is worth several AA/artillery pieces - whether 2 or 5 or 10 or 20 I couldn't say, but definitely more than just one.

    FYI!
    Last edited by TWC_Flug; Dec-21-2021 at 18:41.
    System: Microsoft Windows 10 Pro 64 bit, 10.0.18362 N/A Build 18362, 20,437 MB |
    ASUS GeForce GTX 1060 3GB | Intel Core i5-2500 Quad-Core Processor 3.3 GHz 6 MB Cache LGA 1155 | Intel DB65AL motherboard | ARCTIC Freezer i11 CPU Cooler | SVGA 500 watt power supply | Microsoft Sidewinder 2 Force Feedback joystick

  2. #2
    ATAG_Colander's Avatar
    Join Date
    Nov 2011
    Location
    Bir Tawil
    Posts
    10,609
    Post Thanks / Like
    Total Downloaded
    120.90 MB

    Re: What causes servers to lock up/max out/stop functioning

    My two cents...

    The server needs to update O objects. This means that as O increases, the CPU needs for each game tick also increase hence, at a certain high O, the server will not be able to keep up (depending on the CPU).
    Then there are P players. The status of the updated objects O and the status of each player P need to be sent to the other players. Because of this, the network, after a reaching a high enough P will also have issues.

    Now, each P will send updates to the server which, in turn, will send updates to the other P's. This means that you have an exponential function in the number of updates the server must handle with increasing P.


    In short, every object, player, ship, gun, bullet etc that moves, fires, dies, etc, will have to be processed by the CPU and then sent to all the players P. This means that is not as simple as "do not add more than 5 guns" or "do not add ships". There is no rule as any increase anywhere can potentially make the server hit the wall.

    Now, to make things more complicated, the players have different internet speeds and qualities. This means that the server might have to retry sending updates and whatnot.



    PS: I'm hungover so this might not make any sense

  3. #3
    Team Fusion Artist's Avatar
    Join Date
    Mar 2010
    Posts
    2,555
    Post Thanks / Like
    Total Downloaded
    319.55 MB

    Re: What causes servers to lock up/max out/stop functioning

    What Colander wrote.

    And also: All these active objects constantly update their "awareness bubble" independent of there being any player on the server or not: Vehicles and trains compute their paths, check for obstacles, collisions, etc.. AA and artillery scan their 'bubble' for something to attack. There's a whole world of activity. Hence the 25% base load that you've observed.
    Last edited by Artist; Nov-01-2021 at 16:44.

  4. #4
    ATAG Member ATAG_Torian's Avatar
    Join Date
    Sep 2011
    Location
    Newcastle, Australia
    Posts
    1,563
    Post Thanks / Like
    Total Downloaded
    108.53 MB

    Re: What causes servers to lock up/max out/stop functioning

    Quote Originally Posted by TWC_Flug View Post
    You can add 150 aircraft and remove them and again, no noticeable difference.

    But surely if they are bombers they have shooty things bristling all over them. We've all had the joy of a hundred laser guided guns lighting up while trying to do slash attacks on a large bomber formation. In my own experience with mission building there are definite limits as to the number of AI aircraft, particularly bombers, that I can host on my pc. Now I realise my pc is nothing comparable to what u likely have but proportionally wouldn't the same apply ? Certainly your observations about AA/ships/vehicles/trains and cpu load are absolutely valid but AI aircraft not having much impact seems counter intuitive. Perhaps they don't add much load until they go into action and suddenly a hundred gunners wake up ?
    Anyway, a good post and a worthwhile discussion.
    System specs:
    Gaming laptop: Metabox P750ZM, CPU Intel i7-4790K @ 4GHz, GTX 980m with 8Gb DDR5 VRAM, 16Gb Dual channel DDR3 RAM, 2 x Samsung EVO 840 500Gb SSDs, Win10 Home 64bit.

    Gaming PC: Asus Rampage III Gene mobo, i7 960 3.2Ghz cpu (running at 3.8Ghz), Noctua dual HS & fan cpu cooler, 24GB Kingston 1600mhz triple channel ram, Samsung EVO 500GB SSD, Corsair 750w psu, EVGA GTX 1070 6GB vram vid card, Windows 10 Pro 64bit.

  5. Likes ATAG_Marlow liked this post
  6. #5
    Supporting Member Tibsun's Avatar
    Join Date
    Feb 2016
    Location
    Austria
    Posts
    908
    Post Thanks / Like
    Total Downloaded
    220.96 MB

    Re: What causes servers to lock up/max out/stop functioning

    Quote Originally Posted by Artist View Post
    What Colander wrote.

    And also: All these active objects constantly update their "awareness bubble" independent of there being any player on the server or not: Vehicles and trains compute their paths, check for obstacles, collisions, etc.. AA and artillery scan their 'bubble' for something to attack. There's a whole world of activity. Hence the 25% base load that you've observed.
    Sad to see that also servers cannot really take advantage of mutiple cores, I always thought that those things like computing paths and checking for obstacles and so on could be computed on a 2nd core/thread.
    Atleast we can take away, that if a single core is utilized to almost 100%, things could improve using cpus with faster single core performance.

  7. Likes Baffin liked this post
  8. #6
    Supporting Member Baffin's Avatar
    Join Date
    Apr 2013
    Location
    Northwestern Virginia, USA
    Posts
    1,908
    Post Thanks / Like
    Total Downloaded
    127.26 MB

    Re: What causes servers to lock up/max out/stop functioning

    Quote Originally Posted by Tibsun View Post
    Sad to see that also servers cannot really take advantage of mutiple cores, I always thought that those things like computing paths and checking for obstacles and so on could be computed on a 2nd core/thread.
    Atleast we can take away, that if a single core is utilized to almost 100%, things could improve using cpus with faster single core performance.
    Do most of you feel that performance may increase if we select no Hyperthreading? I'm not sure if my system will allow single core only, but I can stop this Intel multithreading in the BIOS.

    Would doing so impact other players or just myself?
    Windows 10 Pro 64-bit, ASUS TUF X299 MARK 1 (LGA 2066 R4), 2x 1 TB Samsung M.2 SSD 970/960 EVO. Intel Core i7 7800X, overclocked CPU: 4.4 GHz, using TPUII BIOS feature. Air Cooling (GPU:<70°/CPU:<80°) with Hyper 212 Cooler w/ 2 fans. 32.0GB RAM at 2772MHz (16-17-17-36), LG 55" 4K OLEDC7P TV, 2-way SLI using NVIDIA GeForce GTX 1080 Ti's (MSI) overclocked: +80/+250 (MSI Afterburner) using NVIDIA color settings: Highest-8bpc/YCbCr444-Limited. Realtek High Definition Audio, Sony Surround amp w/ optical cable for 5.1 speakers, Ear Buds from Motherboard for Discord/TeamSpeak3. ASUS Blu-Ray drive BW-12B1ST, TrackIR5, Buttkicker Gamer 2, Thrustmaster Warthog, 2x Saitek X-52 (Buttons & Gear), Gear-Falcon Trim Box, Thrustmaster TPR Pendular Rudder Pedals. Voice Activated Controls.

  9. #7
    Supporting Member Tibsun's Avatar
    Join Date
    Feb 2016
    Location
    Austria
    Posts
    908
    Post Thanks / Like
    Total Downloaded
    220.96 MB

    Re: What causes servers to lock up/max out/stop functioning

    Quote Originally Posted by Baffin View Post
    Do most of you feel that performance may increase if we select no Hyperthreading? I'm not sure if my system will allow single core only, but I can stop this Intel multithreading in the BIOS.

    Would doing so impact other players or just myself?
    Never go single core, have you even checked how many threads of your cpu are being used in the OS and how many in the game?
    If each has some work, even just 20%, it would mean this 20% will be shifted to the single core being already overwhelmed and running at 80-100%.
    It means it's already not keeping up and you want to shove more work on it, you're better off not messing around with cores and threading if you don't know what you're doing.

    If you need game performance as a CloD player buy a graphicscard and if the fastest graphicscard isn't enough for other games, you can afterwards get a new cpu (with all it's necessary parts).
    Very fitting, Intel just released new cpus with a new architecture, big power cores with smaller economic cores with high single core performance for the power cores.
    The i5 12600k is roughly 80% faster than your 7800x in single core. Quite an new astonishing milestone for Intel. Current GPUs also have an incredible increase in performance, but are very expensive - still duo to "shortages"

    Pretty sure servers could make good use of the new Intel 12000s cpus

  10. #8
    Veteran Combat pilot
    Join Date
    May 2015
    Location
    Kansas City, Missouri area
    Posts
    460
    Post Thanks / Like
    Total Downloaded
    84.81 MB

    Re: What causes servers to lock up/max out/stop functioning

    Quote Originally Posted by ATAG_Torian View Post
    But surely if they are bombers they have shooty things bristling all over them. We've all had the joy of a hundred laser guided guns lighting up while trying to do slash attacks on a large bomber formation. In my own experience with mission building there are definite limits as to the number of AI aircraft, particularly bombers, that I can host on my pc.
    It's not so much that AI aircraft (and live pilots, too) don't affect the server load, but rather I think we all pretty much understand that both AI and live aircraft do in fact affect server load and we also understand about how many aircraft the server can support - 100 to 150-ish, more or less, depending on your server CPU, internet connection, etc.

    The point is, these other shoot-ey things are an almost independent factor to the number of live/AI aircraft, that is easy to overlook. Especially it is easy to overlook that there are numbers of shoot-ey things beyond AA guns - and that on bring the server to a complete stop with a fairly strong threshold effect. That is to say, you have 2000 shoot-ey things and you add say 20 more - suddenly you've gone from nicely functional server to a completely dysfunctional server that no one can even join.

    This, to me, is somewhat unexpected.

    Also the fact that you can keep an eye on CPU load as a decent approximation of server load or overload is a useful thing to know. Obviously the number of live/AI aircraft figures into the server's CPU load.
    Last edited by TWC_Flug; Nov-29-2021 at 17:57.
    System: Microsoft Windows 10 Pro 64 bit, 10.0.18362 N/A Build 18362, 20,437 MB |
    ASUS GeForce GTX 1060 3GB | Intel Core i5-2500 Quad-Core Processor 3.3 GHz 6 MB Cache LGA 1155 | Intel DB65AL motherboard | ARCTIC Freezer i11 CPU Cooler | SVGA 500 watt power supply | Microsoft Sidewinder 2 Force Feedback joystick

  11. #9
    Veteran Combat pilot
    Join Date
    May 2015
    Location
    Kansas City, Missouri area
    Posts
    460
    Post Thanks / Like
    Total Downloaded
    84.81 MB

    Re: What causes servers to lock up/max out/stop functioning

    By the way, another possible cause of "unexpected" server failure is if the AI aircraft ever get into giant battles with each other.

    It's one thing to have 75 AI on one side and another 75 on the other side, all flying various patrols and routes widely dispersed around the map.

    And it's quite another thing if a bunch of those all meet each other and start fighting. If even say 20 or 30 or 40 of these get into a giant furball you'll likely see the server brought to its knees - that's one of the things I've noticed, anyway, though I'll confess I haven't tested it with extreme thoroughness. But it's the sort of thing that makes a good degree of sense - the server is having to fly all of those planes against each other with rather fine control, aim and shoot, handle multiple weapons per aircraft shooting, etc, compared with what it takes to fly a straight line at 7000 meters with no enemy in sight.

    (Just from a server and campaign design standpoint there is little point in having AI on AI battles in some far corner of the map, even if they do nothing more than add 0.01% server load. They are still pointless and don't add anything of value to the campaign, if there are literally no breathers around to even observe or notice what is happening. So when we noticed this problem happening we just set out to avoid it as best we could, for all those reasons, including maybe it helps the server remain more responsive.)
    System: Microsoft Windows 10 Pro 64 bit, 10.0.18362 N/A Build 18362, 20,437 MB |
    ASUS GeForce GTX 1060 3GB | Intel Core i5-2500 Quad-Core Processor 3.3 GHz 6 MB Cache LGA 1155 | Intel DB65AL motherboard | ARCTIC Freezer i11 CPU Cooler | SVGA 500 watt power supply | Microsoft Sidewinder 2 Force Feedback joystick

  12. Likes Artist liked this post
  13. #10
    Veteran Combat pilot
    Join Date
    May 2015
    Location
    Kansas City, Missouri area
    Posts
    460
    Post Thanks / Like
    Total Downloaded
    84.81 MB

    Re: What causes servers to lock up/max out/stop functioning

    > Do most of you feel that performance may increase if we select no Hyperthreading?
    I did a little reading on this and my understanding is that, as Tibsun said, allowing hyperthreading is almost always advantageous.

    The CPU percentages you see if you have "logical cores" are somewhat misleading. As in, let's say you have a CPU with four real cores and 8 total cores including the logical cores.

    So if you turn hyperthreading off and run a single core-type process you will see that it takes up 25% CPU.

    Now turn hyperthreading on and run the same process - you will see it now takes 12.5% CPU.

    But . . . in reality those are both the same amount of CPU. Your process would run the same speed, reach the same benchmark speeds, whatever, in both situations.

    So even though 12.5% CPU looks like have as much, and you would imagine it would be half as fast, it just isn't. It's actually using one core to the full extent possible, so the 12.5% really is equivalent to 25% with hyperthreading turned off.

    The improvement with hyperthreading, though, is that the CPU can handle multiple demands better. It has literally 2X as many places to put threads, whether they are threads from the main program (launcher -server in this case) or just the various random things the pop up in the windows environment. It can even run threads, especially brief ones, on the other side of the core where the main program is running, because hyperthreading sort of fits the extra logical core into CPU cycles that aren't being used anyway.

    Anway, long & short is that hyperthreading improves performance in pretty much every conceivable situation but the way CPU usage is reported in percentages makes you think that maybe that isn't the case.
    System: Microsoft Windows 10 Pro 64 bit, 10.0.18362 N/A Build 18362, 20,437 MB |
    ASUS GeForce GTX 1060 3GB | Intel Core i5-2500 Quad-Core Processor 3.3 GHz 6 MB Cache LGA 1155 | Intel DB65AL motherboard | ARCTIC Freezer i11 CPU Cooler | SVGA 500 watt power supply | Microsoft Sidewinder 2 Force Feedback joystick

  14. #11
    Team Fusion ATAG_Oskar's Avatar
    Join Date
    Nov 2017
    Location
    Canada
    Posts
    783
    Post Thanks / Like
    Total Downloaded
    837.84 MB

    Re: What causes servers to lock up/max out/stop functioning

    If you use the Resource Monitor you can see the CPU load broken out by core.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •