The development of the AI avatar began in May 2024, and Microsoft introduced Avanade to join as a development partner the following month because this project demanded a short timeframe and development with new technology. “We turned to Avanade, experts in our technology with extensive knowledge," says Mr. Takashi Okawa, Partner Solution Architect for AI & Azure Architect Department in the Partner Technology Division, Partner Business Group, Microsoft.

To provide visitors with an immersive experience of AI technology in the VIP room at Mizuho PayPay Dome Fukuoka — rather than just using the latest generative AI — Avanade integrated various technologies to develop an AI avatar that performs real-time actions as if a person was actually there. The first step in the project was to clarify the specifications of the AI avatar, which were determined from the following four perspectives:

1. The data accessible to the AI avatar
2. The user interface (UI) to be implemented
3. The triggers for the avatar’s responses
4. The Azure functions to be involved

These aspects define the knowledge the avatar has, how it interacts with people and the triggers for its actions. They also determine what the AI avatar can do, how it behaves and the methods and approaches for its implementation. Meanwhile, the SoftBank HAWKS proceeded with selecting candidates to serve as the AI avatar.

“Initially, we considered featuring active players from the SoftBank HAWKS. However, with the season underway and their busy schedules, it was challenging to arrange their participation," reflected Mr. Hiroyuki Ishida, who is the Assistant Manager of Section 3 in Sales Department 1, Sales Division, Business Management Office of the Fukuoka SoftBank HAWKS. This led them to look for a well-known figure among the SoftBank HAWKS' alumni, ultimately deciding to select Mr. Igarashi. In June 2024, an educational video and audio were recorded in Tokyo. The total recording time was approximately two hours, with the video lasting about 30 minutes and the audio about 15 minutes.

Mr. Okawa from Microsoft says: "When creating a realistic avatar of a real person, it is common to use 3DCG modeling software to convert the person into CG. However, this method requires the expertise of designers proficient in 3DCG production, often resulting in prolonged production timelines."

Microsoft's AI avatar leverages video as training data, significantly reducing the learning phase compared with traditional methods. Mr. Okawa explains: “The person being converted into an avatar requires only a few hours to record the training data, for both audio and video. After recording training data, the model's training in the cloud takes about an hour for audio alone, and three to four days when video is included. To prevent unauthorized learning, the system features a requirement for the person providing training data to read aloud ‘I have approved this.’ Without this confirmation, the learning will not proceed.”

During development of the AI avatar, five consecutive one-week sprints were conducted, and the first release was achieved in just a month and a half from the beginning of specification planning — an astonishing pace.

However, the process was not entirely without its challenges. The biggest hurdle in the development of the AI avatar was the delay when using external data. Hiroshi Senga, Senior Director at Avanade, says: “We initially acquired the game data live from the data feed provided by Data Stadium Inc. using a ‘pull"=’ method, where the AI avatar would fetch the data. This caused a delay of up to one minute, resulting in delays in game commentary as well. To resolve this issue, we switched the data-retrieval method from pull to push (where Data Stadium sends the data). Additionally, by streamlining processes and tuning the generative AI for faster processing, we successfully reduced the delay to about 10 seconds. I believe that using real-time data and streaming data for RAG (Retrieval-Augmented Generation, a method of utilizing generative AI with external knowledge) is quite rare. The delay is as fast as, if not faster than, the broadcasts of specialized sports video-streaming services.”

To create an avatar that closely resembles Mr. Igarashi's persona and characteristics with a more realistic understanding of baseball, we performed delicate tuning multiple times, such as the structure and composition of the input data. “Such tuning is essential for maintaining audience engagement. Avanade achieved this level of precision in just one week," says Mr. Nishikawa.

The AI avatar was designed with a special focus on creating a user-friendly interface. According to Hiroshi Senga from the Avanade team: “Avanade's user-experience specialist team was involved in shaping its user interface, meticulously designing it to be clear and easy for the visitors.”

The Fukuoka SoftBank HAWKS Corp. (referred to as the SoftBank HAWKS) is a professional baseball team belonging to the Pacific League of Nippon Professional Baseball (NPB), based at the Mizuho PayPay Dome Fukuoka in Fukuoka City. Within the dome, on the fourth floor of the Super Box (the luxury suite area), there are 16 exclusive VIP rooms called the “Microsoft Premium Suites,” which feature an “AI avatar” of Ryota Igarashi, a former SoftBank HAWKS player who now acts as a baseball commentator.

Mr. Takuya Nishikawa, Vice President, Deputy Head of Sales, Fukuoka SoftBank HAWKS, explains the background behind the development of this AI avatar: “SoftBank HAWKS has long had the desire to create an iconic space within the dome to honor SoftBank, our parent company. SoftBank owns a facility called the EBC (Executive Briefing Center) for offering solutions to their clients, and we had an idea to set up a new satellite booth for it in the dome. An AI avatar was created as part of this project, with the intention of using Microsoft’s latest AI technology through the strategic partnership between SoftBank and Microsoft.”

"Various efforts have been made toward improving stadium DX in the past, but they all stayed as tests and were not implemented. The reason is that no matter how advanced the technology, it did not reach a level where the regular visitors to the dome could enjoy and be fully satisfied. This time, we are confident that customers will find it interesting and that it will contribute to improving customer satisfaction," says Mr. Nishikawa.

The AI avatar was officially released in July 2024. It responds to user questions, provides brief comments on player information based on the game situation, and offers cheering messages for the SoftBank HAWKS — all in real-time with gestures from Mr. Igarashi. At the bottom of the screen, users can also see the information of the day’s starting lineup and a "HAWKS Quiz" with trivia questions about the team.

"The AI avatar created by Avanade has very natural intonation, speed and blinking, with accurate lip-syncing to the voice, making it feel as if Mr. Igarashi himself is right in front of us,” says Mr. Ishida. We have set about 10 gesture patterns, and it moves naturally at the right moments according to the game situation."

It is located in one of the 16 rooms of the Microsoft Premium Suite on the fourth floor. Given the limited opportunities to interact with AI in Fukuoka, many people are surprised to see Mr. Igarashi's AI avatar. Inside the Microsoft Premium Suite, staff trained in EBC hospitality guide visitors through the AI avatar experience. As a satellite of the EBC, this suite primarily serves corporate clients.

"The first client was impressed and called it 'amazing.' This led to referrals spreading and inquiries from many other companies. We are already fully booked for the entire next season. The AI avatar has introduced a new way to enjoy baseball games at the stadium and has greatly contributed to lead generation for the entire SoftBank Group," says Mr. Nishikawa. "The Microsoft Premium Suite will be the gateway to experiencing the AI avatar in other rooms. Looking ahead, the team aims to integrate it with the customer IDs from the SoftBank HAWKS official app — enabling food and merchandise orders, as well as providing tailored responses to clients' preferences."

“Avanade leveraged Microsoft’s latest AI technology, along with bringing on dedicated design staff to create a high-quality user interface,” says Mr. Okawa. “By entrusting this project to Avanade, we feel it became a truly excellent project."

When everyone and everything is connected by data, Avanade can help turn your data, analytics and AI ecosystem into your biggest advantage.