Add Smart Attacking to Your PySC2 Agent

In my last tutorial, we added a Q-Learning table to our PySC2 agent, however the attack coordinates were pre-programmed, meaning the agent really only worked on the Simple64 map and was prone to a stalemate when enemy units or buildings were outside the attack radius of the units once they reached the attack coordinates.

In this tutorial we will add smart attacking to the agent so that it can learn where to attack based on the detected position of enemy units or structures.

Please note that this tutorial picks up where the last tutorial left off. You should copy the final code from the tutorial in order to follow these steps.

To my friends from the Blizzard/Deep Mind AI Workshop at BlizzCon, this is the first tutorial that covers the details from my presentation.

1. Set Up

To get started, let’s make a few adjustments to our existing code that we will use in the rest of the tutorial.

First, let’s add a constant:

Next, let’s rename our class:

Don’t forget the parent step method call:

If you followed some early versions of my previous tutorials it also seems some people were having trouble with undefined properties, so let’s cascade our init method to the parent class (this has now been fixed in the previous tutorials):

We are also going to do some fancy things with the mini-map, so let’s rename the existing transformLocation method to transformDistance and add a new transformLocation method as follows:

The new transformLocation method will convert absolute x and y values based on the location of your base, instead of just the distance.

Since we have renamed our transformLocation method to transformDistance we should update our actions that used the old transformLocation method:

Now we can start adding the smart attack functionality.

2. Alter the Attack Action

Previously we had only a single “attack” action, we are going to replace that action with several actions so that our bot can attack one of several points on the mini-map.

Replace the existingsmart_actions definition with the following:

Your smart_actions variable now contains an action to attack every single x and y coordinate combination on the mini-map.

Now that the bot can select the attack action from the list of possible actions, we need to make sure it executes the attack correctly.

Insert the following code before the entire IF condition segment:

What we do here is detect the presence of the underscore in the smart_action variable, since no other actions contain an underscore we can assume it is a mini-map attack action. Next we split the smart action to get the x and y coordinates.

Now we can update the attack condition as follows:

Note that we cast the x and y values to integers as they are still strings from the split command.

Now would be a good time to test it out and see how it performs:

python -m pysc2.bin.agent \
--map Simple64 \
--agent attack_agent.AttackAgent \
--agent_race T \
--max_agent_steps 0 \
--norender

It may look like your bot is doing nothing, because in a way it is! Due to the extremely large number of possible actions, most of the time it will be trying to attack a location even if it has not selected a unit. We will deal with this issue in the next step.

3. Simplify the Actions

Although our mini-map is a 64x64 grid, the attack radius of marines is large enough that you only need a 4x4 grid to effectively cover all of the Simple64 map, so we are going to reduce our attack points as follows:

Reduced attack points for the Simple64 map

This can reduce your attack actions to 16 combinations with the following code:

Now that each square actually covers a 16x16 grid of sub-squares, we move the attack coordinates of the middle of the larger squares by shifting the values by 8 on each axis. This provides better coverage of the map than sticking to the top-left corner of each square.

Give it a test run and notice how much faster your bot discovers the ability to build the supply depot and barracks.

4. Add Enemy Positions

After a while your bot may figure out that it should attack the general area of your enemy’s base, however if units are out on the map it may also learn to attack in that area regardless of whether or not there are enemy units there. This is because your reward is associated with the position rather than the presence of enemy units.

We can improve this by adding enemy positions from the mini-map to the current_state variable.

Now you can run the agent again, with enough games it will begin to learn to attack coordinates that contain enemy units or structures.

5. Simplify the State

One of the problems with so many enemy coordinates is your bot can take a really long time to explore all of the possible options. Just as our mini-map can be reduced to 16 squares for the attack space, we can also reduce it to 16 squares for the state space, reducing the exploration time:

Enemy position grid for the Simple64 map

In order to do this we check each coordinate within each square, and if any coordinate contains an enemy unit the entire square is marked as “hot”.

Now run your code and see that the attack component of your bot is much smarter!

6. SCV Attack Cheat

One thing that may frustrate you about your current bot is it still spends a lot of time sending SCVs out to attack the enemy. In a future tutorial we will cover a more refined rewards system that can discourage this sort of behaviour, however for now you can simply cheat and disallow the ability to attack with SCVs by replacing this code:

With this:

Once this code is in place I have found my agent is able to defeat the enemy within a very small number of games.

All code for this tutorial is available here.

In my next tutorial you will find out how to use sparse rewards to train your agent.

If you like this tutorial, please support me on Patreon. Also please join me on Discord, or follow me on Twitch, Medium, GitHub, Twitter and YouTube.