My in-console Civilization 6 SIEM
For reasons I can’t remember, I found myself browsing the Civilization 6 games files and came across it’s pretty comprehensive in-game logging.
In exploring the logs, I start to discover some potentially useful information. I see how it generates the map, the results of combats and even the starting and finishing co-ordinates of each unit, after each turn.
Knowing barely anything about the game, and struggling to even win against the medium difficulty, I began to wonder if this data could help me win.. or cheat.
I realize that building a SIEM for the game would be a great weekend project and maybe help me cheat mywin a game. I would need to do what any other Cyber Security Engineer would do in real life which is to conduct OSINT, model the threat landscape and alert on threats.
OSINT and Threat Analysis
Fortunately for Civilization players, but unfortunately for me, the game already alerts the player to many different useful in-game metrics. However, I’m really hoping to find data in the log files that you wouldn’t find in the game’s UI.
I end up finding mountains of possible alert worthy data but after some research, a couple of hours getting my butt kicked by the medium difficulty AI, and also for the sake of scope creep (which I’m good at moving), I conclude that I would like to alert on:
- When other players spawn new military units (including Barbarians)
- When these military units are approaching my cities
For this functionality, I work out that I’ll need ‘tailer’ to read the last lines of log files and I’ll also need to make the most of threading to read multiple files concurrently.
The basic structure of my code will be to create a function that monitors csv files and one to monitor log files. Then a third ‘monitoring’ function that handles the threading of each of those functions.
‘Threat Landscape’
Now that I know what data I have available to me and what I’d like to be alerted on, I thought it would be fun to plot the literal ‘threat landscape’. Not something usually done in this manner but assuming the ‘Duel’ map size of 44x26 and one other enemy AI, as well the Barbarians and other city states, I plot out the entirety of the map using the co-ordinates I find in ‘Lua.log’ with help from NumPy and Pandas:
This is a full drawing of all starting positions of all civilizations as of turn 1. One last assumption would be that these co-ordinates are also the ones the player decides to settle their capital city on. Just because the game spawns on you a particular tile, doesn’t necessarily mean you always settle there.
Forgive me for breezing past how I did this, this was the only time I allowed myself to stray out of scope. I’m very conscious that I get nerd sniped but I’m always looking to practice and improve my data analysis and visualization skills with Python. I might return to this write up and walk through how I did it but for now if you’re interested - check out the Github here.
Alerting on Barbarian Spawns
Back on track again, I start with when new Barbarian units are spawned and where. There is a very handy but also messy ‘Barbarians.csv’ which tells me when and where a new Barbarian military units are spawned:
With some regex and a little data-frame indexing I end up with exactly what I need:
In this process, I learn that Barbarians.csv isn’t created until the end of turn 1, or maybe start of the second turn. AI players only operate in the time between player turns and of course there’s no predecessor to turn 1. Frustratingly, this means we miss the first alert on ‘UNIT_SPEARMAN’ because the SIEM starts monitoring the file only once it’s found it, where the log entry has already been placed.
To manage this, I reused the same logic in my for loop when monitoring the file, to just read the second the second row of the csv ad hoc:
if csv_file_path == 'C:\\Users\\User\\AppData\\Local\\Firaxis Games\\Sid Meier\'s Civilization VI\\Logs\\Barbarians.csv':
# Find first log of barbarian unit
# Skip the header line
next(csv.reader(file))
# Read the second line
second_line = next(csv.reader(file))
new_barb_string = new_barb_unit_pattern.search(second_line[2])
unit_string = new_barb_string.group(1)
print(f"-- New Barbarian Military Unit spawned: {unit_string} at{second_line[3]}!")And the result:
Now when monitor_csv_file() detects ‘Barbarians.csv’, it reads and prints the second line and then proceeds it’s normal duties of listening for any new log entries to the file. Relying on it to detect the file and start listening to it, before it gets a chance to record it’s first few logs was never going to happen!
Alerting on AI unit spawns
To track when the other player (or ‘Major Start’ as the logs call it) spawns new units, there’s an ‘AI_CityBuild.csv’ which tracks each cities production queue. It’ll actually tell me when the AI submits something to it’s build queue, regardless of whether it’s a building, improvement or unit. It’s an incredibly rich set of logs and knowing when the enemy has even submitted something to it’s build queue could be a huge advantage. For now, I think knowing when an AI unit has spawned into the game is enough of an advantage for me but knowing when it’s submitted to the queue might be something I add later down the track if the medium difficulty is still stomping me.
For this alert, it’s a similar process to before but with a few differences. I’m listening to a different CSV file and as mentioned, there’s an if statement to ensure that I’m only alerting on units that have actually spawned into the game (i.e where Construct = ‘ COMPLETED’ in ‘AI_CityBuild.csv’) and not just where they’ve been ‘ SUBMITTED’ to the City build queue.
if csv_file_path == 'C:\\Users\\User\\AppData\\Local\\Firaxis Games\\Sid Meier\'s Civilization VI\\Logs\\AI_CityBuild.csv':
if row[5] == " COMPLETED":
new_unit_string = unit_name_pattern.search(row[6])
new_unit_city = city_name_pattern.search(row[2])
unit_string = new_unit_string.group(1)
city_string = new_unit_city.group(1)
print(f"\n-- New AI Unit spawned: {unit_string} in the city of {city_string}!") Printing turn numbers
After setting up alerts for new units, I see it can get quite noisy and looking back at the alerts, there isn't as much chronological context as I would like.
I figure declaring the start of each turn would give me a good chronological idea of how the game’s developing in hindsight. I pick out ‘AI_Planning.csv’, a file that I know tracks each turn (rather than only when a unit is spawned) and quickly configure it to print ‘TURN X:’ whenever a new turn is started.
This all turned out much harder than expected..
The interesting thing I learnt is that, as mentioned earlier, the AI ‘plays’ in the time between turns. Only after the player selects ‘Next Turn’ does the AI do anything in that turn before proceeding to the next. So technically from the players perspective, it’s in the current turn that the player will be notified of the previous turn’s events.
This felt a little counter-intuitive at first but understanding that although the human player might play a 100 turn game, there’s really 200 turns from the game’s perspective (assuming one other AI player). Where human players make their moves on the whole numbers (x.0), and the AI players make their move in the x.5 between. However, in a game where there’s 1 human player, 1 other CPU player, a Barbarian’s civilization and even the multiple city states, the game in reality has many hundreds of turns.
To visualize this, this was taken whilst in turn 3 — note that there are no alerts in the current turn 3:
Next, this was taken in turn 4, where an alert has appeared for turn 3:
The alert only appeared to the player as they were on their way to turn 4. This is confirmed in the logs:
HOWEVER! Sporadically and very annoyingly, there was a bug where alerts where appearing under the current turn.
In the same game, only a move later in turn 5, you can see a new Spearman spawned in co-ordinates 1–0 in turn 4:
But this time, the alert in the console has told me that it’s happened in turn 5, the turn I’m currently in, which we know to be impossible with how the in-game turn system works, nothing should be spawning in until the theoretical turn 5.5:
Originally with how sequential Python is - I thought by declaring the logic that handles the game-turn notification before the logic of the unit spawn alerts, would mean it would be the first text printed to the console, followed by what units have spawned. However when this didn’t work, I realized that of course the game itself has it’s own order of operations, where not all files are updated at the same time. They themselves would follow a specific update order, that could mean the alerts don’t make chronological sense to the player as we see here.
In this bug, the ‘Barbarians.csv’ I am using to alert on is getting updated AFTER AI_Planning.csv (The file I’m using to notify the player of the current turn) causing the Barbarian alert to appear late. But only sometimes it seems, in the image above the bug clearly didn’t present between turns 3 and 4.
Inspecting ‘Barbarians.csv’ and ‘AI_CityBuild.csv’, I see that from turn 4 to 5, ‘Barbarians.csv’ was updated with a new Spearman, and as expected ‘AI_Planning.csv’ was as well. However I notice that ‘AI_CityBuild.csv’ recorded no new logs between turns 4 to 5.
My theory to explain this is that sets of log files in the game are updated concurrently or at least in sets, rather than in a specific order each time. Since ‘AI_CityBuild.csv’ and possibly other log files had nothing to log between turns 4 and 5, ‘AI_Planning.csv’ is updated fractions of second earlier than it was on previous turns - and annoyingly in my case before ‘Barbarians.csv’ - causing this bug where it looks like units are spawning in the current turn when they are not.
In short, the bug is caused not by ‘Barbarians.csv’ being read late, but ‘AI_Planning.csv’ being updated early, and then ultimately messing up my alert timeline.
To remedy this I put a simple 0.2 second sleep timer in the turn notification to ensure that all the AI’s actions are captured and printed before the next turn number, and started to pray that I could get a test game with the perfect conditions to test this bug fix.
if csv_file_path == 'C:\\Users\\User\\AppData\\Local\\Firaxis Games\\Sid Meier\'s Civilization VI\\Logs\\AI_Planning.csv':
time.sleep(0.2)
if int(row[0]) > turn_number:
turn_number = int(row[0])
print("------------------------------------")
print(f"\nTURN {turn_number}:") Sure enough, but maybe not soon enough - Also on turn 5, where no logs were made to ‘AI_CityBuild.csv’ in turn 4:
Success! Although the chronology of alerts wouldn’t be a problem in a traditional SIEM, knowing this bug existed in specific edge cases would have kept me up at night.
Alerting on approaching units
Now that I have the functionality of knowing when there’s new enemy units on the board — My next goal is to alert on when these units are approaching my city. For this, I need to define a perimeter around it and then alert when an AI unit enters this perimeter.
..Is it a stretch to say this sounds like a Firewall?
I reuse the same logic to plot out my starting position on the ‘Threat Landscape’ figure and use those co-ordinates to build the perimeter off.
Using those previously assumed co-ordinates, I’ve setting an X and Y perimeter of +5/-5:
# Set perimeter
upper_x_perimeter = int(player_city['X_coords']) + 5
lower_x_perimeter = int(player_city['X_coords']) - 5
upper_y_perimeter = int(player_city['Y_coords']) + 5
lower_y_perimeter = int(player_city['Y_coords']) - 5‘AStar_GC.log’ is a file that at the start of each turn, lists every unit’s starting co-ordinates, and then the co-ordinates they end the turn on. If those co-ordinates match these conditions:
# Check if finish co-ordinates are within perimeter:
if lower_x_perimeter < unit_x_finish < upper_x_perimeter and lower_y_perimeter < unit_y_finish < upper_y_perimeter:
# Ensure I don't alert on my own units
if city_name != player_city['name']:
print(f"\n-- {unit_name} from {city_name} is nearby! ({unit_x_finish}, {unit_y_finish})") Then an enemy unit is nearby! And I want to know about it:
My alerting is now complete! I added some quality of life features, such as adding more context to my own X, Y co-ordinates, knowing the nearby enemies co-ordinates but not your own was more intimidating than helpful. Also declaring my own and the enemies starting civilization’s name as well as making further distinction between turns. Here is how the first few turns look when the game and notebook are running in tandem:
Conclusion
Writing in hindsight, this project has been the perfect microcosm of Cyber Security and the role a Cyber Security Engineers plays. There is so much more data I’d love to dive into and alert on but as much of a valuable learning curve it’s been writing my own custom SIEM from scratch, next I’d like to ingest in these logs into Microsoft Sentinel and write the equivalent KQL queries — so stay tuned for that!
Jacob Lummus - jclummus.w@gmail.com
