I utilized some of my Google-Fu and came up with this and this . It's the same person on two different websites and both solutions seem to include a fresh install of Windows 10 64bit. I'm not sure what you're using right now but I would not use this solution first since Windows 10 is not free anymore.
Are you using any sort of advanced launch parameters on Arma 3? I see that you have 4096MB of VRAM set aside for Arma 3. I suggest turning off that parameter and see what happens. There was a small blip at the bottom of that setting that states if you put it over 2GB of VRAM, it can have catastrophic results.
I would also look into updating your video drivers via GeForce Experience, having the most up-to-date drivers can do wonders for your computer.
Also, check out this thread , it's the same guy but different solution. Making sure that you have disabled your integrated graphics from your i7-8700K and have everything render from your 1080ti can fix the memory issue since integrated GPUs only have 500MB (if that) of VRAM.
This is also a good read regarding 32-bit and 64-bit operating systems and the game. It's from 2008 so it might be kind of worthless. Do make sure you're running 32-bit Arma 3 for your 32-bit OS and 64-bit Arma 3 for your 64-bit Arma 3.
Some guy from this thread gave a detailed graphic regarding RAM usage in vanilla Arma 3 and it leads me to my next question, how many GB of RAM are you running?
| In use (Compressed) | Available | Committed | Cached | Paged pool | Non-paged pool
=============================================================================================
Idle | 3.0 GB (106 MB) | 12.9 GB | 3.6/18.3 GB | 2.0 GB | 350 MB | 115 MB
A3 | 5.9 GB (1.3 GB) | 9.5 GB | 11.4/18.3 GB | 2.9 GB | 428 MB | 143 MB
---------------------------------------------------------------------------------------------
Usage | 2.9 GB (1.2 GB) | 3.4 GB | 7.8/18.3 GB | 0.9 GB | 78 MB | 28 MB
I threw a lot at you but I hope at least some of it is helpful. I'm no tech genius by any means so there's a lot that I do not know and I just skimmed the threads to get to the good parts. I wish you the best of luck!