This is a translation from the Sega Tech Blog. In this post, Atlus programmer Takahiro Nowatari explains the system built during the development of Persona 5 Royal to automatically play through the game and find bugs. It’s interesting because he talks about things from an engineering perspective -- he goes over the problem requirements, time and resiliency constraints, result data, and the pros and cons of his approach. Enjoy it!

Note: This translation is for fan use only, and may not be accurate. Be cool, don’t repost without asking, link to this page, and credit Brando!

Original post (Japanese): https://techblog.sega.jp/entry/2020/09/25/100000

Finding Bugs Via Autoplay During Persona 5 Royal’s Development

2020-09-25

Introduction

Hi everyone, I’m Nowatari, a programmer at Atlus, and I’ll be your author for this entry of the Sega Tech Blog.

This time, I’d like to talk about how we implemented a system to automatically play and find bugs in Persona 5 Royal (P5R) while the game was still in development, and the results that it gave us. This post will be more about techniques devised for debugging work, rather than information about implementing the actual game.

That being said, I’ll be talking about the contents of P5R quite a lot, so if you haven’t played the game yet, I think it will help deepen your understanding if you’re able to do so before reading this.

P5R - ペルソナ５ザ・ロイヤル

[English site: https://atlus.com/p5r/ ]

The Motivation for Implementing Autoplay

When I joined the company many years ago, bug detection was done by playing the game with a video recorder hooked up to the development hardware, and when a bug appeared, writing down in detail the sequence of events leading up to it in a bug report (on paper) and submitting that as a record of what happened.

These days, video recording features are integrated into the development tools themselves, and bug reports are managed via a Web-based ticketing system, so the whole environment has changed considerably since back then.

Not just methods to report bugs, but methods to find them have also changed. Before, testers would spend hours glued to the screen, manually playtesting to uncover bugs, but now there’s been a trend towards automating the things which can be tested mechanically.

The average playtime to complete P5R is approximately 100 hours, so even if someone devoted all of their working hours to just playing through the game, it would still take close to three weeks for them to reach the end.

That being the situation, I wanted to somehow increase the amount of hours we could spend playtesting the game, even just a little bit. If possible, I wanted to create some mechanism that would allow us to validate the game all the way through to the ending. From those ideas, I decided to implement a system that would automatically play through the game and test it for us, without human intervention.

On Implementing Autoplay

There were a few problems standing in the way of implementing autoplay.

No dedicated hours were allotted for this project, so we couldn’t ask others to do work for it.
For the same reason, we couldn’t spend a lot of time fixing and fine-tuning the autoplay system itself.

About #1, for example, even if I wanted to ask the designers or planners to do something to help support this system, their hands were already full with tasks from their own areas of responsibility, so I couldn’t ask them to take on additional work.

For #2 as well, I myself was responsible for the game’s everyday life and dungeon parts (the parts where you control the player and move around freely), so it wasn’t like I’d have plenty of time after making the autoplay system to update it in response to specification changes from the design and planning side.

Under these circumstances, the final system:

Should rely on specialized data as little as possible, since there was no time to create it.
Should be able to handle specification changes and adjustments [to the main game], since there was no time for fine-tuning it.

I started working on it with this kind of concept in mind.

Also, as implementation objectives:

It should be able to play the game automatically from beginning to end, without human interaction.
By adding random elements, it should be able to check as many areas of the game as possible.

Aiming for something that could achieve these goals, I started creating the autoplay system.

The Implementation

Before explaining the implementation details, allow me to briefly explain some of the systems in this game.

One of the characteristic features in the Persona games is the calendar system.

The calendar system

In the game, everything takes place within a one year timespan, so as the calendar days go by, the game moves forward. In the extreme case, even if the player lazily just went straight home and slept every day, they would still get closer to the ending.

Good things come to those who wait?

...But of course, just doing that is no way to make progress. The player does have to beat dungeons and defeat bosses at fixed intervals in order to beat the game. Also, just sleeping the days away is no way to find bugs either, so I also wanted to make the autoplay system trigger various in-game events as much as possible. For that, we needed a mechanism to make the player character go not just to bed, but to all sorts of locations.

So the question here became: how do we make the player character move to target locations and progress through the game?

For the dungeon parts, progressing through the game means solving puzzles and gimmicks and arriving at the boss. For the daily life parts, it means talking to NPCs to trigger events, going to the batting center, and activities like that. By completing these kinds of events, the calendar days pass by and the player can reach the ending without getting a game over.

As for the mechanism for moving to a target location, the very first thing that came to mind was replaying someone’s actual playthrough data, or possibly designating those sorts of movements using development tools. But, implementing that wasn’t possible because of the circumstances I mentioned earlier (no time to prepare special data for autoplay purposes).

Because of that, I decided to implement a rule-based movement system which wouldn’t require any extra data to be created.

The core of this rule-based logic is a mechanism to search for and activate nearby event hits.

Here, the term “event hit” refers to a region in-game that the player can access and trigger some event from.

Examples of event hits

Talk Hit for talking to NPCs
Check Hit for checking treasure chests and doors
Go-to Hit for going to a different area
...etc

By using these event hit regions as the target locations, we’re able to make the player character move to them and trigger the aforementioned events that move the game forward, instead of just running around blindly.

In our implementation, we randomly search for locations the player can reach by moving in a straight line, without being blocked by obstacles, focusing on the area directly in front of them (the fan-shaped area in front of the player in the following screenshot).

Searching for event hits

However, with just this, the movement system had some trouble when there weren’t any event hits nearby, so we added one more rule.

That additional rule involves the use of a heatmap, tracking where the player has already been within a certain time period (the red-outlined blocks in the following screenshot).

The heatmap

We add this time-limited wandering history to the area the player has already walked around (in this case, using 100cm cubes). By doing so, we’re able to establish this rule: places with no heatmap are places the player hasn’t walked around (at least not recently).

With this rule, if we’re not able to find any event hits, we search the surrounding space for areas with no heatmap (blocks), and by using those areas as target locations, the player character is able to seek out and walk around places they haven’t been yet.

Example movement during autoplay

We move towards these unexplored areas that have no heatmap, without getting stuck even in winding passageways, and when we find an event hit we go and examine it. The general process should be clear now, I think.

Unexplored areas have a high probability of leading to event hits we haven’t triggered yet, and when we do trigger them, we get more unexplored areas, then we move there and find more unexplored areas... It creates this loop where we’re able to explore broader areas and go further and further, moving completely automatically.

Some other, minor adjustments were also made, but overall the two movement rules mentioned above enable the player character to navigate unexplored areas, and by triggering the event hits found along the way, they’re eventually able to reach the dungeon’s boss.

The daily life parts of the game don’t have one target destination like the final area of a dungeon, but with the previous mechanisms implemented, we made the system able to trigger all sorts of events in those parts as well, by using similar rules for interacting with various NPCs and in-game facilities.

With everything so far in place, we were able to handle the other parts of the game that don’t involve player movement (like battles and dialogue choices) with random input or something like that to progress, and eventually the system was able to reach the game’s ending.

The event hits used as target locations in our autoplay system were already in the game as usable data, so we were able to implement the system so that it didn’t need (hardly) any additional data, and didn’t need fixing after changes were made to the main game.

Because of that, from the middle of development to just before mastering, we started running the autoplay system on a development machine that wasn’t in use (such as when everyone left the office at the end of the day), and even though various changes were made [to the main game], the autoplay system itself was able to keep running with almost no adjustments.

Practical Results

We tested the game using this autoplay system, mainly checking for bugs which could cause the game to hang or make it impossible for the player to progress.

As a result, out of all the bug reports for those highest priority must-fix issues, the autoplay system was able to find about 5% of them.

This number might seem a little low, but considering that it only counts recorded bug tickets, I think the percentage is a little higher if we include other issues that the system discovered midway through development but weren’t recorded as bug reports.

Also, a lot of the bugs found by automatic play involved timing or methods difficult to reproduce by hand, so the fact that it discovered bugs which we couldn’t have predicted was another big win.

In battle, doing an All-Out Attack at a specific timing leaves the player unable to progress
Loading data while the game is saving causes the game to freeze
Doing a persona fusion with a specific timing and procedure causes the game to freeze
Etc...

Starting from the middle of the game’s development, one development machine was always used for the autoplay system, and it ran pretty much 24 hours a day. As a result, when the final development deadline arrived, the system had accumulated over 2000 hours of play time (roughly 250 man-days of work!!).

Summary

We were able to implement this autoplay system in a way that required almost no additional specialized data, and almost no adjustments to address changes in the main game.

It might seem like our implementation method had a lot of merits, but of course it also had some downsides too.

Since we used random elements, it couldn’t check every part of the game exhaustively
Similarly, because we used lots of random operations, it couldn’t check things which required complex conditions to be met
The player character’s movement wasn’t very efficient, so each playthrough took around 300 hours to finish

These are some of the problems and areas for improvement we felt after actually using the system.

As a follow-up exercise, I’d like to think about how to solve these problems and come up with ways to check the game more efficiently, and in further detail.

At Atlus, we’re looking for people interested in tackling these kinds of challenges. If you are interested, please have a look at the page linked below.

Brando Life

Pages

Saturday, April 3, 2021