LIVE: Proximal Policy Optimization (PPO) Trains Sean in Street Fighter III through PCSX2

One PPO policy plays Sean in Street Fighter III: 3rd Strike inside PCSX2. There is no combo script. Each step the network picks a button hold or a whole special, the emulator plays it, and the reward comes from the vitality change in PS2 memory. Two matches run at the same time and share that one policy. Each match is its own PCSX2 process, its own PINE socket, and its own round-start savestate: slot 1 is Sean vs Necro (SA2 Slam Dance), slot 8 is Sean vs Yun (SA3 Genei Jin). Sean is SA1 Hadou Burst. When a round ends, that window reloads its savestate and the episode starts again. First to 2, 99-second timer, 160 vitality, difficulty 4. The policy sees pixels, not a game-state vector. The window is captured at 200×256, cut in half to 100×128, and kept as a stack of 9 frames. The observation handed to the CNN is 3 channels taken from frames 2, 5, and 8 of that stack, so the net can see motion. Stable-Baselines3 wraps that as a CnnPolicy. The action space is discrete, 63 choices. Indexes 0–43 are the original holds: directions, punches, kicks, and the throw pairs. Indexes 44–62 are Sean’s specials, each one a motion the wrapper plays for him: Dragon Smash, Ryuubi Kyaku, Tornado, Zenten, Sean Tackle, Hadou Burst, and the EX versions. Forward and back are flipped from the two fighters’ X positions, so the same action still faces the opponent after a side switch. A basic hold is kept for 4 emulated frames. Each piece of a special is kept for 4 frames too. Buttons do not go through the keyboard. A patched PCSX2 ANDs port 0’s pad read with a word at EE 0x01FFF000. The trainer writes that word every frame over PINE. Bits are active-low. Zero means no override. Vitality, timer, and round count are read from PS2 memory, also over PINE. The reward is 3 × damage dealt − damage taken, then multiplied by 0.001. If neither fighter is hit for more than 20 steps, each extra step costs 0.1 before that scale. A win pays 3 × 160^((life+1)/161), so leftover vitality is worth more than a bare KO. A loss is the same curve on the opponent’s remaining life, negated. A timeout where both sides have equal vitality is a draw and costs 80 before the scale. A KO has to read the same way for 2 steps before the episode ends. PPO is on CPU. Rollout length is 512 steps per environment, batch size 512, 4 epochs. Gamma is 0.94. The learning rate moves from 2.5e-4 down to 2.5e-6, and the clip range from 0.15 down to 0.025. Entropy coefficient is 0.005. An episode is cut at 1800 steps if the round is still going. Checkpoints are every 10,000 steps. The run resumes a 510,000-step policy whose head only had the 44 button holds. The 19 new special rows were copied from his most-used attack and shifted down so those specials start at about 22% of his choices. The target is 5,000,000 steps. The page on the left reads the training log. It is not part of the policy. On the board while this was written: 576,560 / 5,000,000 steps, 141 rounds, 56 wins, 85 losses. Necro 35–34. Yun 21–51. #PPO #ReinforcementLearning #PCSX2 #StreetFighter #StreetFighter3 #3rdStrike #nullmicgo