Abstract: In this talk, starting from the research of natural language processing technology, we will comprehensively analyze the application of natural language processing in robots, such as multi-round dialogue problems in human-robot interaction, AI writing creation, and a comprehensive analysis of natural language processing on robots, analysis of application cases of language processing on robots. Then, we will present a comparison study of retrieval augmented generation (RAG), supervised fine-turing, (SFT), and offline reinforcement learning to mitigate hallucination for multimodal large language models (LLMs). Final, we point out the future research directions of natural language processing technology in human-robot interaction.

Bio: Dong-Yan Huang (M’96-SM’05) received her bachelor and master degrees from Xi’an Jiaotong University, Xi’an, China, in 1985 and 1988, respectively, and the PhD degree in Syst`eme Physique et M´etrologie-Communication & Electronique from the Conservatoire National des Arts et M´etiers Paris (CNAM), France, in 1996. She is now a Principal Scientist at UBTECH Robotics Corp. From Dec. 1997 to Nov. 2002, she was a Senior Research Engineer at the Institute of Microelectronics, Singapore. Before that, she was a postdoctoral researcher at the UFR de Math´ematiqueset Informatique, Universit´e Paris Descartes, France. From Dec. 2002 to 2019, she was a Senior Scientist at the Institute for Infocomm Research, Singapore. Her research focuses on machine learning, pattern recognition, affective computing, automatic speech recognition, text-to-speech synthesis, voice conversion, computer vision, dialogue system, talking head, human-machine interaction, robotics and embodiment intelligence. She has authored more than 100 publications in peer-reviewed journals, and conference proceedings. She was solicited and co-chaired for ASMMC from 2015 to 2021. She has been serving as the program committee for several international conferences in the areas of signal processing, speech processing, multimedia, human-computer interaction, affective computing and intelligent interaction. She was the chair of the IEEE Singapore Sensor Committee Sub-Committee (2016-2018), the chair of the WIE (Women in Engineering) group (2006-2008). She led a team to implement online and offline speech technology on Cruzr, Walker robots and a series of educational products. Her team’s works on digital emotion won the awards of A*STAR’s 30 Most Impactful Innovations & Inventions over Three Decades (2021), and P&G Connect + Develop Open Innovation Solutions Award on Digital Insights 2020, the first prize in the 2011 INTERSPEECH Speaker State Challenge Sleep Competition and the first prize in the EmotioNet Challenge.

Abstract: In today's landscape of natural language processing (NLP) and speech processing, developing applications often begins with fine-tuning a foundation model. However, teaching a foundation model like LLaMA new skills is not as straightforward as it seems. Introducing new capabilities can often impair their original functions, a phenomenon known as catastrophic forgetting. While experience replay is a common solution, the need for training data for the foundation models poses challenges for continuous training. This talk will delve into recent research on fine-tuning language models, including their spoken counterparts, focusing on preserving their initial capabilities.

Bio: Hung-yi Lee is a professor of the Department of Electrical Engineering at National Taiwan University (NTU), with a joint appointment at the Department of Computer Science & Information Engineering of the university. His recent research focuses on developing technology that can reduce the requirement of annotated data for speech processing (including voice conversion and speech recognition) and natural language processing (including abstractive summarization and question answering). He won Salesforce Research Deep Learning Grant in 2019, AWS ML Research Award in 2020, Outstanding Young Engineer Award from The Chinese Institute of Electrical Engineering in 2018, Young Scholar Innovation Award from Foundation for the Advancement of Outstanding Scholarship in 2019, Ta-You Wu Memorial Award from Ministry of Science and Technology of Taiwan in 2019, and The 59th Ten Outstanding Young Person Award in Science and Technology Research & Development of Taiwan.