Practice Public Speaking with VR

What do you specialize in?

My field of expertise is Applied Linguistics, which is a vast and extensive field of study. I focus on English education, media application and languages from a sociological point of view. Like I have said, it is a very extensive field. The areas I specialize in are English education using media and multimodality.

What is multimodality?

Multimodality comprehensively analyzes human interactions, communications and all symbolic methods other than words and spoken language. For example, when we talk, there are elements other than words and the literal meanings exchanged. For example, our face conveys information through facial expressions, we use gestures, the sound of our voice fluctuates, our speaking speed, and intonation are all elements that convey information.

These elements have been regarded as peripheral components of language called paralanguage. Suppose you are talking on an online platform like Zoom, multimodal analysis may include the virtual background, clothes you are wearing, and if you're using an avatar, what kind of an avatar you are using as targets of analysis. For example, is your avatar a cat or a human? These factors also form meaning. Multimodal analysis is a field that analyzes multiple elements in an integrated manner.

That sounds exciting! Why did you choose this area of study?

Originally, I was interested in the meaning-making of how humans communicate. If you climb a mountain with your family and you arrive at a very beautiful view together, you might say, "Oh, it's beautiful!" "Yes, it is!" That is a type of casual everyday conversation. However, when I was small, I was anxious because when I said, "It's beautiful," there was a likely possibility that what I see is different from what the other person saw. I worried if I could articulate the thought and if the message was properly conveyed. On the other hand, if the other person understood what I wanted to say, just as I meant it, why was the communication successful?

Due to the nature of my parents' work, I grew up in an environment where I had close contact with people from different regions of the world, such as South Korea, China, English-speaking countries, and other foreign places. Back then I didn't understand languages other than Japanese, so I had no idea what they were talking about. I didn't know what they were saying, but I could see their expression, and heard the tone of their voice. So, through observing the multimodal aspects, I could tell, "this person is very happy right now!" I became aware of communication that was formed from something other than words and I became interested in the limits of language.

When I entered university, I met a professor of cognitive linguistics and enrolled in their course. It's a field that touches on things I was concerned about. Communication is so dangerously ambiguous, but somehow, we understand each other and are happy to talk with one another. That is when I found the name to the question that I had.

Cognitive linguistics deals with the linguistic phenomena of how humans perceive and verbalize the world. It encompasses human cognition so it also studies meaning-making, and how people interpret the world. Linguistics is one research field that deals with how humans understand the world.

I then went to study at a graduate school in England, and my supervisor there specialized in multi-modality. There, I found what I had been looking for all along! This is what I have been wondering since I was a child! The part of human cognition that is not just language but is comprehensively represented by paralanguage, symbols and signs. I found that there is this thing called multi-modality. It really clicked. So, by chance, I met two professors who clicked with my pre-existing interests from childhood, and it has allowed me to stay on this path.

You seem to have had a uniquely inquisitive mind like a researcher when you were a child.

(Laughs) Well, I have been told that as a child, no matter what I was told, I immediately asked "Why?"

We can see some information on your website that is intriguing! Could you tell us what kind of research are you doing now?

While I analyze the text and voice content every time, I am now focused on multi-modality. Virtual reality technology has developed so much in recent years that it easily matches the concept and research area of multi-modality. Partly due to that, I have been analyzing the behavior of public speaking speakers through presentations and speeches from a multi-modality perspective. I look at specific targets of behavior. I have found some characteristics of effective speakers. I was able to clarify these traits to some extent.

The funny thing about speeches and presentations is that there is no one, how-to method on what is best. Rather, I found the “don’t do-s”. These “don’t-do-s” were gathered through data and I wanted to make a public speaking teaching material for Japanese people.

I spent about two years planning how to make it, and around the second year into the planning, I realized virtual reality would be perfect. VR can be used to support eye-contact and track facial movements. I wanted to create a system that allows people to learn through experience, and I have received grants for scientific research from MEXT Kakenhi for several years. I have also received various budgets to proceed with its development.

Are you involved in product development with a company?

Yes, I have been outsourcing a part of the development process to a company that develops various VR systems. We have been working together.

According to your research results, incorporating the “don’t do-s” leads to more effective VR training programs.

In Japan, students don't really learn public speaking skills in school. Even if it is taught in school, students are not given much time to practice. For example, in many cases, they only speak in front of the class once or a few times each. So, many students don't make eye contact with the audience while speaking. There are a lot of students who come to university, and to my class, that speak very quickly when they get nervous, and hardly make eye contact especially if it is a presentation in English.

They might look at the presenter's tool on the computer the whole time or the screen behind them instead of the audience. They tend to have only two positions of their notes in front and the screen behind them instead of looking at the entire audience. For the audience, they may feel that the speaker is neglecting them, but often times the speaker feels they are making eye-contact because they looked up from the computer (to the screen). However, they are unaware of its ineffectiveness. It is difficult to reflect on our own actions due to the nature of public speaking.

I was keenly aware of how difficult it was to practice alone, because you cannot observe yourself. Through behavioral analysis of good speakers, such as information on how well they maintained eye contact, we found they look at the audience often, and rhythmically, left, front, and right, in all three directions. Through my research I found that people that were ranked as having great engagement cross-culturally, no matter who was evaluating them. Japanese, American or British people, all thought that eye contact maintained for a certain duration and moved rhythmically felt best.

I've gradually come to understand things like how the audience can become engrossed in listening and how a speaker can foster this through good eye contact, and the speed of speech should not be too fast or slow. I have come to understand the parameters of good public speaking multimodalities, so I made a program in the VR space. In the beginning the person moves their face as if they were chasing a target. If you are able to do it, you will be given points, so the person becomes aware how much they are able to do. If you look down the whole time, your score will be low, so speakers can self-reflect, and understand what they can do to improve. They will realize that they have to have the speech memorized somewhat, or they cannot look up from their notes.

Even if it is not completely memorized, they must have each of the topics and the order in their mind. Once the speaker realizes that they must have aspects of the speech memorized, the next part of the VR training is about the timing and spacing of the eye contact. They will be given a score for this. So, eye contact and the speed of the speech are the main aspects of review.

This VR system seems very useful in the field of education.

Thank you. I would like to use it more in my classes.

You have acquired patents based on your research results. How was the application process?

Yes, the VR teaching materials are patented. Initially, it was because a professor of psychoacoustics from the Faculty of Design joined this VR project. He had already obtained several patents by this time and advised me that I should file too for the technological results of our research. Before then, I didn't consider that to be an option. I didn't really know how. I was lucky that an office at Kyushu University handles such requests.

The Academic Research and Industrial Collaboration Management Office called AiRiMaQ can determine if an employee at Kyushu University's invention qualifies for patent filing. The experts on intellectual property will help register and file for the employee invention. Filing patents uses a lot of technical language so close communication with the AiRiMaQ and patent attorneys was needed. The process also took a long time. I think the patent was registered two or three years after I first sent the university's invention registration application.

That is amazing you have a patent.

Well, yes, it was nice to know that the novelty of the research and outcome has been recognized. There was also a comment from the office that they hope it is actually used in the field, so I hope that there are enough people that work in education that would like to use it. Then, a company might be interested in creating it as a product. It was nice that the patent was registered so the conversation around product development would go smoothly.

What is joint research with companies like for researchers?

If the agreement is such that only one party benefits from the collaboration, it will not go well. Both parties need to contribute, and proceed with that intention. Proper consensus building is necessary. I was also involved in the JST program for the social implementation of technology seeds. In addition, I attended the entrepreneurship education program.

The goal-setting of a company and the goal-setting of an academic researcher are pretty different, and although there is some overlap, there are differences. For example, in general, a new business’s goal setting is to be profitable within 5 years. The sense of speed in which to make a profit is a difference in goals. For an academic researcher, the research timeline may be 10 or 20 years from now, depending on the research theme and fields.

It is nice that the results of research reach many people, but there are various hurdles and differences to overcome. What was your project in Taiwan?

I had an opportunity to collaborate with National Taipei University of Technology. I taught students there for some specialized courses (such as Research Methods in Applied Linguistics) and research co-supervision, and my students here in Japan were also taught by the faculty in Taipei Tech as co-supervision. Many thanks to the collaborating professor, we exchanged lots of research information. This exchange first began as a program for SENTAN-Q.

The research exchange led to cooperation with Taipei Tech's international program even after the completion of SENTAN-Q. In that program, their students interested in international exchange learned about Kyushu University's campus and education programs in English and participated in real-time online campus tour sessions with me.

I guided the students through our campus via live-streaming. I prepared a lot, looking into the history of the design of the campus and the characteristics of our international education programs and such things. They listened attentively to my introductions of our university and my research topics. I was impressed by their English skills and pointed questions regarding my VR projects. One unexpected funny surprise was that one of the most active discussions happened when I was showing them the snack section of the Co-op, (who were kind enough for me to stream from there also). Students would tell us if specific snacks were available there in Taiwan, or if they were more expensive in Taiwan, such as the chocolate bar Black Thunder. They also had questions about the coffee machine at the store. (laugh) I hope the program enhanced their interests in studying at Kyushu University and one day they participate in the exchange study-abroad program.

Do you have any advice for students who want to go into your field of Applied Linguistics?

If you are a high school student who wants to pursue applied linguistics, I recommend that they study a wide range of theories during their undergraduate period. Linguistic theory has a long history, perhaps it is an academic field with the longest history in the world. If you go back, it goes to the Greeks and their ideas on rhetoric. The long history has made it evolve over time and has created many genres. You can audit classes of these various studies.

Linguistics are divided into roughly five areas, phonetics, phonology, morphology, syntax and semantics. If you know what type you are most interested in, it would be appropriate for you to apply to a university that has a professor that is an expert in that area so that they could adequately guide you. For example, I was interested in meaning-making, so semantics was my main area. A university might not have an expert of an area of linguistics you are interested in. However, you can read entry-level books so you cover the range, even if you don’t have a professor that teaches the area of interest, it will help build a basic understanding of the areas. If you would like to go on to become a researcher, it’ll be vital that you have an integrated knowledge of the field.

Was machine translation made possible with linguistics?

The study into grammar and syntax, yes, however I'm not a specialist in machine translations. I did have a student who researched translation to incorporate the multimodalities. They hoped to get more accurate translations based on the speaker’s intention in applications that use real-time voice translations such as Zoom. They looked at the pitch and other multimodal aspects.

Your students are also doing interesting research.

Yes, my students are amazing. In human interactions, the intention is important. In some cases, the person themselves are not aware of their intentions when it comes to the voice. For the research, a third party listened to the objectively recorded data and judged what the person meant by what they said, and the speaker themselves also documented what they meant. Sometimes the intention of what is said is different, so even in actual communication, such errors can occur, and it causes misunderstandings. When researching how hard it is to make meaning with various multi-modal elements, I am amazed at what humanity is able to accomplish every day!

What areas of linguistics do you think will be exciting in the future?

The relationship between robots, AI, and humans. We are already entering a time when we might not be certain of what we are interacting with. Are they humans in the Metaverse or systems like Chat GPT? You can communicate with others regardless of age, gender, nationality, etc., and you can freely change your appearance every day in the Metaverse. I hope that more positive interactions and way of life can be achieved with this possibility.