Pukyong National University

What if a blind person could see objects by sound?
Research Story of Prof. Kim Chae-gyu of Artificial Intelligence Research Center at PKNU

△ Professor Kim Chae-gyu. ⓒPhoto by Lee Sung-jae (PR Team)

The performance of research professor Kim Chae-gyu (age 68, kyu0707@pknu.ac.kr) of the Artificial Intelligence Research Center at Pukyong National University attracts public attention.

In his 40s and 50s, he led the ICT field at the National Electronics and Telecommunications Research Institute (ETRI), the best information and communication technology (ICT) research institute in Korea, and is now showing off his own know-how to the Pukyong students.

○ He shares 'ICT know-how' built up over his lifetime to the youth of the Pukyong.

As one of the frontiers who opened the path of information and communication technology (ICT) in Korea, what path has he followed and what kind of research is he doing now?

Professor Kim is from Korea University (Dept. of Mathematics). He received a master's degree in computer science from the University of Technology Sydney, Australia and a doctorate in computer science from Woolongong University.

Starting out as a researcher at the Korea Institute of Science and Technology (KIST) in 1977, he moved to ETRI in 1998, and for 16 years until 2014, he served as the head of the ETRI Computer Software Research Center, the head of the Digital Home Research Center, the head of the research strategy division, and the head of the convergence technology research division.

○ The relationship with Pukyong National University began in 2014… Active as a visiting professor in the Department of IT Convergence and Application Engineering

It was in 2014 that he had a relationship with Pukyong National University. He became a research professor at the Pukyong National University Industry-Academic Cooperation Foundation. For three years from 2017, he was active as a visiting professor at the Department of IT Convergence and Applied Engineering at Pukyong National University as the 'Inviting and Utilizing Professional Experienced Personnel Support Project' of the National Research Foundation (NRF) of Korea.

As a visiting professor, he opened three subjects ▷Introduction to IT Convergence Application, Software System Safety Engineering, and Software Engineering in the undergraduate program, also he opened and lectured three subjects ▷ Introduction to Advanced IT Convergence, Special Topics in U-city Engineering, and Multimedia Software Engineering in Graduate School with the project cost of the National Research Foundation of Korea (180 million won).
 
In the process, he conducted research projects (430 million won) with students (30 undergraduates, 3 graduates) from ETRI, such as ▷development of face modeling and transformation technology that is linked with interactive media creation service platform, ▷sensory replacement research trend survey, and ▷Research projects such as sensory replacement research trend survey and development direction research.

At the time, students actively published papers (5 thesis) in academic journals, received the Best Paper Award from the Korea Information Processing Society (KIPS) and the Korea Multimedia Society (KMMS), and the President's Award from the Capstone Design Competition on campus. In addition, they participated in summer and winter vacation internships operated by ETRI and exchanged with ETRI doctoral researchers. Three students were enrolled in the master's course at Ulsan University of Science and Technology (UNIST) and the University of Science and Technology University (UST), and one at the Ph.D. program at Pukyong National University.

Since last March, he has served as a research professor at the Artificial Intelligence Research Center at Pukyong National University.

What kind of research is he doing here?

○ What if a blind person could see things by sound?

The 'Multi-sensory substitution that enhances human sensory perception ability and convergence technology-related development project', which Professor Kim is collaborating with ETRI at the Artificial Intelligence Research Center, is drawing public attention.

This is a research to develop digital future technology using artificial intelligence. To develop an alternative technology for artificial intelligence sense to overcome the limitations (time, space, intelligence, language, and body) that humans face.

Professor Kim said, "Sensory replacement is a technology that transforms primitive stimuli and transmits them through a sense organ other than the original one. The final goal of this research is to make the stimulus transformed through the device reach an externalized state that feels like a primitive stimulus from the outside." For example, it refers to allowing the visually impaired to identify objects through a device that visualizes sound.
 
"This study not only allows people with impaired senses to experience lost sensations, but also allows them to sense information that humans have not previously accepted," additionally he explained, "Developing sensory augmentation and replacement devices capable of detecting these new sensations will allow us to supply more information to the brain through unusual pathways. If we let our brains recognize more parts of the world, we will experience another world of superpowers."

○ What if I appeared in TV series 'Descendants of the Sun' instead of Song Hye-kyo or Song Joong-ki?

'Development of face modeling and transformation technology linked to interactive media creation service platform' is also his main research project.

This research is a technology that replaces one's face with a face in a specific media through facial transformation technology. It aims to develop and test models that can act in real time by deforming faces by creating deep learning-based facial transformation models and 3D masks.
 
When Prof. Kim was at ETRI, "We developed 'COX', which is an interactive video creation platform that provides interaction between users and media for the first time in the world," he said. It is an interactive media platform that enables interaction between actors and audiences in video.

This platform provides a variety of interaction editing functions such as ▷a branch function for selecting a storyline, ▷a function for changing the actor's face in a video, ▷a push function for sending a message to a video, ▷a tele function for making a call with a video, and ▷a connection function for connecting the video object and the web. In addition, it provides services to be used in interactive art and movies, experience-based machines and functional motion recognition games, and interactive advertisement production.

Professor Kim said, "COX not only provides viewers with the feeling of seeing and feeling the video, but also providing the feeling of participating in the story progression of the video and creating the result and experiencing it directly."

He said, "By installing Face Swapping technology on the ‘COX’ platform, I want to commercialize the world's first interactive media platform through social networks to quickly provide services that reflect the needs of the market, and I want to contribute to the creation of a smart media service ecosystem through cooperation with local governments and companies." <Pukyong Today>


△ Professor Kim Chae-gyu, a research professor, is demonstrating 'Face modeling and transformation technology linked to interactive media creation service platform'.