Shane Moon

I'm Shane (Seungwhan) Moon, and I am a Research Scientist at Meta (Facebook) Reality Labs working on Multimodal Learning and Conversational AI for the AR/VR applications. Before that, I completed my PhD at Carnegie Mellon University (CMU) where I worked with Prof. Jaime Carbonell on Proactive Transfer Learning and Multimodal Learning with various applications in NLP and CV.

Some of my non-academic pursuits include photography, music, skiing, cycling, etc.

Email: me at shanemoon dot com

Also at: Google Scholar | LinkedIn | Twitter

Work Experience

Publications

AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Seungwhan Moon*, Andrea Madotto*, Zhaojiang Lin*, Tushar Nagarajan*, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, Anuj Kumar
(*: joint first authors)
arXiv, 2023.
( PDF, BibTeX )

IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
Seungwhan Moon*, Andrea Madotto*, Zhaojiang Lin, Alireza Dirafzoon, Aparajita Saraf, Amy Bearman, Babak Damavandi
(*: joint first authors)
EMNLP, 2023.
( Code & Data, Ambient AI Workshop @ ICASSP'23, PDF, BibTeX )

SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams
Te-Lin Wu, Satwik Kottur, Andrea Madotto, Mahmoud Azab, Pedro Rodriguez, Babak Damavandi, Nanyun Peng, Seungwhan Moon
ACL, 2023.
( PDF, BibTeX )

Tell Your Story: Task-Oriented Dialogs for Interactive Content Creation
Satwik Kottur*, Seungwhan Moon*, Aram H. Markosyan, Hardik Shah, Babak Damavandi, Alborz Geramifard
(*: joint first authors)
EACL, 2023.
( PDF, BibTeX )

Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks
Pedro Rodriguez, Mahmoud Azab, Becka Silvert, Renato Sanchez, Linzy Labson, Hardik Shah, Seungwhan Moon
EACL, 2023.
( Code & Data, PDF, BibTeX )

Embodied Executable Policy Learning with Language-based Scene Summarization
Jielin Qiu, Mengdi Xu, William Han, Seungwhan Moon, Ding Zhao
arXiv, 2023.
( PDF, BibTeX )

A Survey of Datasets, Applications, and Models for IMU Sensor Signals
Aparajita Saraf, Seungwhan Moon, Andrea Madotto
ICASSP Ambient AI, 2023.
( PDF, BibTeX )

Navigating Connected Memories with a Task-oriented Dialog System
Seungwhan Moon*, Satwik Kottur*, Alborz Geramifard and Babak Damavandi
(*: joint first authors)
EMNLP, 2022.
( Code & Data, PDF, BibTeX )

Normalized Contrastive Learning for Text-Video Retrieval
Yookoon Park, Mahmoud Azab, Seungwhan Moon, Bo Xiong, Florian Metze, Gourab Kundu, Kirmani Ahmed
EMNLP, 2022.
( PDF, BibTeX )

KETOD: Knowledge-Enriched Task-Oriented Dialogue
Zhiyu Chen, Bing Liu, Seungwhan Moon, Chinnadhurai Sankar, Paul Crook, William Yang Wang
NAACL, 2022.
( Code & Data, PDF, BibTeX )

Overview of Situated and Interactive Multimodal Conversations (SIMMC) 2.0 Track at DSTC 10
Satwik Kottur*, Seungwhan Moon*, Alborz Geramifard, Babak Damavandi
(*: joint first authors)
AAAI DSTC10 Workshop, 2022.
( PDF, BibTeX )

SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations
Satwik Kottur*, Seungwhan Moon*, Alborz Geramifard, Babak Damavandi
(*: equal contributions)
EMNLP, 2021.
( Code & Data, PDF, Presentation, BibTeX )

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
Zhiyu Chen, Honglei Liu, Hu Xu, Seungwhan Moon, Hao Zhou, Bing Liu
EMNLP, 2021.
( Code & Data, PDF, BibTeX )

Continual Learning in Task-Oriented Dialogue Systems
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Zhiguang Wang
EMNLP, 2021.
( PDF, BibTeX )

Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba and Pascale Fung
EMNLP, 2021.
( Code, PDF, BibTeX )

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue
Hung Le, Chinnadhurai Sankar, Seungwhan Moon, Ahmad Beirami, Alborz Geramifard, Satwik Kottur
ACL, 2021.
( Code & Data, PDF, BibTeX )

An Analysis of State-of-the-Art Models for Situated Interactive MultiModal Conversations (SIMMC)
Paul Crook*, Satwik Kottur*, Seungwhan Moon*, Ahmad Beirami, Eunjoon Cho, Rajen Subba and Alborz Geramifard
(*: equal contributions)
SIGDIAL, 2021.
( PDF, Presentation, BibTeX )

Adding Chit-Chat to Enhance Task-Oriented Dialogues
Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, Claire Cardie
NAACL, 2021.
( Code & Data, PDF, BibTeX )

Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking
Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Stephen Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho and Rajen Subba
NAACL, 2021.
( Code, PDF, BibTeX )

Situated Interactive Multimodal Conversations (SIMMC) Track at DSTC9
Paul A. Crook*, Satwik Kottur*, Seungwhan Moon*, Ahmad Beirami, Eunjoon Cho, Rajen Subba, Alborz Geramifard
(*: equal contributions)
AAAI DSTC9 Workshop, 2021.
( SIMMC Track Page (Code, Dataset), PDF, Slides, BibTeX )

Overview of the Ninth Dialog System Technology Challenge: DSTC9
Gunasekara et al.,
AAAI DSTC9 Workshop, 2021.
( DSTC9 Page, PDF, BibTeX )

Situated and Interactive Multimodal Conversations
Seungwhan Moon*, Satwik Kottur*, Paul A. Crook^, Ankita De^, Shivani Poddar^, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, Alborz Geramifard
(*, ^: equal contributions)
COLING, 2020.
( DSTC9 Challenge, SIMMC Track Page (Code, Dataset), PDF, Presentation, BibTeX )

Information Seeking in the Spirit of Learning: a Dataset for Conversational Curiosity
Pedro Rodriguez, Paul Crook, Seungwhan Moon, Zhiguang Wang
EMNLP, 2020.
( PDF, Code & Data, Dataset Viewer, BibTeX )

User Memory Reasoning for Conversational Recommendation
Hu Xu, Seungwhan Moon, Honglei Liu, Bing Liu, Pararth Shah, Bing Liu, Philip S. Yu
COLING, 2020.
( PDF, BibTeX )

OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs
Seungwhan Moon, Pararth Shah, Anuj Kumar, Rajen Subba
ACL, 2019 (Best Paper Nomination).
( GitHub, PDF, BibTeX )

Memory Grounded Conversational Reasoning
Seungwhan Moon, Pararth Shah, Anuj Kumar, Rajen Subba
EMNLP, 2019.
( PDF, BibTeX )

Memory Graph Networks for Explainable Memory-grounded Question Answering
Seungwhan Moon, Pararth Shah, Anuj Kumar, Rajen Subba
CoNLL, 2019.
( PDF, BibTeX )

Active Federated Learning
Jack Goetz, Kshitiz Malik, Duc Bui, Seungwhan Moon, Honglei Liu, Anuj Kumar
NeurIPS Workshop on Federated Learning, 2019.
( PDF, BibTeX )

Federated User Representation Learning
Duc Bui, Kshitiz Malik, Jack Goetz, Honglei Liu, Seungwhan Moon, Anuj Kumar, Kang G. Shin
arXiv, 2019.
( PDF, BibTeX )

Zeroshot Multimodal Named Entity Disambiguation for Noisy Social Media Posts
Seungwhan Moon, Leonardo Neves, Vitor Carvalho
ACL, 2018.
( PDF, BibTeX )

Multimodal Named Entity Recognition for Short Social Media Posts
Seungwhan Moon, Leonardo Neves, Vitor Carvalho
NAACL, 2018.
( PDF, BibTeX )

Learn to Active Learn: Dynamic Active Learning with Attention-based Strategies Selection
Seungwhan Moon, Jaime Carbonell
2018.
( PDF, BibTeX )

Completely Heterogeneous Transfer Learning with Attention - What And What Not To Transfer
Seungwhan Moon, Jaime Carbonell
IJCAI, 2017.
( PDF, BibTeX )

This Malware Looks Familiar: Laymen Identify Malware Run-time Similarity with Chernoff faces and Stick Figures
Eliezer Kanal, Nathan VanHoudnos, Will Casey, David French, Brian Lindauer, Seungwhan Moon, Peter Jansen, Jamie Carbonell, Evan Wright
EAI, 2017.
( PDF, BibTeX )

Proactive Transfer Learning with Heterogeneous Feature and Label Spaces
Seungwhan Moon, Jaime Carbonell
ECML-PKDD, 2016.
( PDF, BibTeX, Poster )

Metaphor Detection Using Topic Transition, Emotion and Cognition in Context
Hyeju Jang, Yohan Jo, Qinlan Shen, Michael Miller, Seungwhan Moon, Carolyn Rose
ACL, 2016.
( PDF, BibTeX )

Cross Modal Content Based Objective for Learning Adequate Multimodal Representations
Seungwhan Moon, Adhiguna Kuncoro, Akash Bharadwaj, Volkan Cirik, Chris Dyer, Louis-Philippe Morency
ICML MVRL Workshop, 2016.
( PDF, BibTeX, Slides )

Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition
Seungwhan Moon, Suyoun Kim, Haohan Wang
NIPS MMML Workshop, 2015.
( PDF, BibTeX, Poster )

Joint Photo Stream and Blog Post Summarization and Exploration
Gunhee Kim, Seungwhan Moon, Leonid Sigal
CVPR, 2015.
( PDF, BibTeX, Extended Abstract )

Ranking and Retrieval of Image Sequences from Multiple Paragraph Queries
Gunhee Kim, Seungwhan Moon, Leonid Sigal
CVPR, 2015.
( PDF, BibTeX )

Metaphor Detection in Discourse
Hyeju Jang, Seunghwan Moon, Yohan Jo, Carolyn Rose
SIGDIAL, 2015.
( PDF, BibTeX )

Proactive Learning with Multiple Class-sensitive Labelers
Seungwhan Moon, Jaime Carbonell
Proceedings of DSAA, 2014.
( PDF, BibTeX, Slides )

Identifying Student Leaders from MOOC Discussion Forums through Language Influence
Seungwhan Moon, Saloni Potdar, Lara Martin
Proceedings of EMNLP, 2014.
( PDF, BibTeX, Slides )

Active Learning with Partially Observed Data
Seungwhan Moon, Calvin McCarter, Yu-Hsin Kuo
Proceedings of WWW, 2014.
( PDF, BibTeX, Slides )

Design and Implementation of the Note-taking Style Haptic Voice Recognition
Seungwhan Moon, Khe Chai Sim
Proceedings of ICMI, 2012.
( PDF, BibTeX, Slides )

Application of Genetic Programming to Verilog Hardware Design
Seungwhan Moon, Paul Booth, Jared Kirschner, Matt Belland
CCSCNE, 2012 (Best Paper Award).
( Abstract )

Education

Awards and Honors

Samsung Fellowship (2013 - 2018)
50,000 ($/Year) for 5 Years of Graduate Studies

LTI Graduate Research Fellowship (2013 - 2018)
Full funding provided by Language Technologies Institute, CMU

Hyundai Best Presenter Award (2016)
Global Top Talent Forum by Hyundai Motors Group

Olin Merit Full Scholarship (2009 - 2013)
4 Year Full Tuition Scholarship (Undergraduate)

Best Paper Award (2012)
Undergraduate Research Competition at CCSCNE 2012

Presidential Scholarship (2008)
Given by the President of South Korea for academic distinction

3rd Place Award (2008)
In the 21st IYPT as a representative of the National Team of Korea

Miscellaneous

Engineering Projects (Archive)

Here is a link to some of my old research projects or projects that I worked on in the past.

Music Compositions

Music is a huge part of my life, and I enjoy both playing and composing music.

Photography

I am also passionate about photography. Check out some of my photos here.

Travel

Some of the places I visited, and some pictures.

Move to top