Health & Human Performance
Digital Healthcare
- Automatic Diagnosis System (ADS) based on natural language
- AI doctor chatbot answering like human with Large Language Model (LLM)
- Automatic pose correction for sports
Our Research
We study how predictive models, language models, and visual information can support healthcare and human performance. Our published work covers prediction-augmented diagnosis, patient-specific medication recommendation, and explainable golf swing analysis.
- Integrating task-specific predictive models with language model reasoning.
- Modeling relationships among diagnoses, procedures, and medication choices.
- Refining human pose estimates and explaining differences in sports movements.
Our Publications
Prediction-Augmented Generation for Automatic Diagnosis Tasks
Abstract summary
Prediction-Augmented Generation (PAG) connects task-specific predictive models with large language models for automatic diagnosis. Rather than relying only on next-token generation or copying an external model’s output, PAG uses predictive results together with language-model knowledge to refine its reasoning. The paper evaluates this approach across diagnosis datasets and introduces a way to assess generated answers in label-based prediction tasks.
Read the original paper Open PDF ↗CafeMed: Causal Attention Fusion Enhanced Medication Recommendation
Kelin Ren, Chan-Yang Ju, and Dong-Ho Lee.
Patient-conditioned causal weights and cross-modal attention model interactions between medical entities.
GolfMate: Enhanced Golf Swing Analysis Tool through Pose Refinement Network and Explainable Golf Swing Embedding for Self-Training
Chan-Yang Ju, Jong-Hyeon Kim, and Dong-Ho Lee.
Pose refinement and interpretable movement representations support golf swing feedback.
Foundational & Influential Papers
Selected papers that shaped the methods and concepts behind this research area.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Connects language generation to retrieved evidence; useful context for understanding external knowledge in language models.
Attention Is All You Need
Introduces the Transformer architecture underlying many modern language and multimodal models.
Large language models encode clinical knowledge (Med-PaLM)
Introduces MultiMedQA and a framework for evaluating medical language model responses, providing a key reference for clinical knowledge and reasoning research.
Audio & Multimodal Intelligence
Audio Intelligence
- Multimodal retrieval for audio-to-text conversion
- Audio generation derived from video content
Our Research
We explore audio representation learning to connect sound, language, and visual context. Our research interests include audio-to-text conversion, multimodal retrieval, and audio generation from video.
- Learning shared representations for audio and natural language.
- Retrieving audio through semantic descriptions and multimodal context.
- Exploring sound generation conditioned on video content.
Foundational & Influential Papers
Selected papers that shaped the methods and concepts behind this research area.
Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)
Large-scale multilingual training for speech recognition and translation.
Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation
Aligns sound and natural-language descriptions for retrieval and audio classification.
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
Generates audio using language-audio representations and latent diffusion.
Computer Vision & Visual Tracking
Multi-Object Tracking
- Robust object detection and efficient tracking algorithm
- Personal information de-identification on image or video sequences
- Face/Person tracking on crowded scenes
Our Research
We investigate robust visual tracking in challenging videos. Our BoT-FaceSORT work combines shot change detection, shared feature memory, and adaptive matching to maintain face identities across frames with abrupt transitions and appearance changes.
- Robust face tracking in unconstrained movie and web video sequences.
- Detection and association under occlusion and appearance variation.
- Privacy-aware visual analysis, including selective face de-identification.
Our Publications
BoT-FaceSORT: Bag-of-Tricks for Robust Multi-Face Tracking in Unconstrained Videos
Abstract summary
BoT-FaceSORT addresses face tracking in videos with abrupt scene changes, occlusions, and changing appearances. It combines shot-change detection, shared feature memory, and adaptive cascade matching to maintain identities across frames. The work also introduces MovieShot, a benchmark built from challenging movie clips, and evaluates the contribution of each component across three tracking benchmarks.
Read the original paper Open PDF ↗Selective face de-identification scheme using multiple face recognition and classification techniques
Sang-Min Bae, Ji-Sung Park, Chan-Yang Ju, and Dong-Ho Lee.
Explores selective face de-identification through recognition and classification techniques.
Foundational & Influential Papers
Selected papers that shaped the methods and concepts behind this research area.
ByteTrack: Multi-Object Tracking by Associating Every Detection Box
Uses lower-confidence detections during association to recover occluded objects and reduce fragmented tracks.
Simple Online and Realtime Tracking with a Deep Association Metric (Deep SORT)
Combines motion-based tracking with learned appearance features to improve identity association through occlusions.
Robotics & Embodied Intelligence
Robotics & Applied AI
- Explainable LLM on Robotics through Thought-Cloning
- RoboNLP: Integrate NLP into Robots
- Simultaneous Localization And Mapping (SLAM) based autonomous driving
Our Research
Our research interests include language integration in robotics and autonomous navigation. Our published SLAM work studies relative coordinate inference using monocular cameras, distance estimation, and trilateration in dynamic environments.
- Relative localization with accessible camera-based sensing.
- Visual understanding in environments containing moving objects.
- Research directions in NLP-integrated robotics and explainable language reasoning.
Our Publications
Efficient Relative Coordinate Inference for Dynamic SLAM Exploiting Monocular Cameras
Abstract summary
This work studies relative position inference for robots operating in dynamic environments. It combines camera-based distance measurement with trilateration, using fixed reference elements to estimate the camera’s location. The approach is designed to work with monocular cameras and explores a lower-cost alternative to sensing setups that require more extensive hardware or computation.
Read the original paper Open PDF ↗Foundational & Influential Papers
Selected papers that shaped the methods and concepts behind this research area.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Transfers knowledge from vision-language pretraining into robot action prediction.
ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM
A reference system for camera-based localization and mapping.
Multimodal Learning & Recommendation
Multimodal Representation Learning & Recommendation
- Attention-guided visual-textual representation learning
- Fine-grained cross-modal feature alignment
- Efficient and personalized multimodal recommendation
Our Research
We investigate how visual and textual information can be aligned to improve personalized recommendation. Our recent MambaRec work combines multi-scale attention, local feature alignment, and global distribution regularization. This research connects multimodal representation learning with efficient information access.
- Learning consistent representations across visual and textual modalities.
- Combining local semantic alignment with global distribution regularization.
- Balancing recommendation quality with representation efficiency.
Our Publications
Modality Alignment with Multi-scale Bilateral Attention for Multimodal Recommendation
Abstract summary
MambaRec learns how visual and textual item features can work together in recommendation. Its attention-guided framework aligns fine-grained features locally and regularizes modality distributions globally. A dimensionality-reduction strategy also addresses the cost of high-dimensional representations. The paper evaluates the combined design on real-world e-commerce datasets, focusing on recommendation quality, generalization, and efficiency.
Read the original paper Open PDF ↗Foundational & Influential Papers
Selected papers on cross-modal representation learning and recommendation.
Learning Transferable Visual Models From Natural Language Supervision (CLIP)
Learns a shared image-text representation through contrastive pretraining, a key foundation for cross-modal retrieval and visual-textual alignment.
LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
A simplified graph model for collaborative filtering that learns user and item embeddings through neighborhood aggregation.
