Over the past few years, the use of the Internet for healthcare-related tasks has grown by leaps and bounds, posing a challenge in effectively managing and processing information to ensure its efficient utilization. During moments of emotional turmoil and psychological challenges, we frequently turn to the internet as our initial source of support, choosing this over discussing our feelings with others due to the associated social stigma. In this paper, we propose a new task of multi-modal medical concern summary (MMCS) generation, which provides a short and precise summary of patients’ major concerns brought up during the consultation. Nonverbal cues, such as patients’ gestures and facial expressions, aid in accurately identifying patients’ concerns. Doctors also consider patients’ personal information, such as age and gender, in order to describe the medical condition appropriately. Motivated by the potential efficacy of patients’ personal context and visual gestures, we propose a transformer-based multi-task, multi-modal intent-recognition, and medical concern summary generation (IR-MMCSG) system. Furthermore, we propose a multitasking framework for intent recognition and medical concern summary generation for doctor-patient consultations. We construct the first multi-modal medical concern summary generation (MM-MediConSummation) corpus, which includes patient-doctor consultations annotated with medical concern summaries, intents, patient personal information, doctor’s recommendations, and keywords. Our experiments and analysis demonstrate (a) the significant role of patients’ expressions/gestures and their personal information in intent identification and medical concern summary generation, and (b) the strong correlation between intent recognition and patients’ medical concern summary generation The dataset and source code are available at https://github.com/NLP-RL/MMCSG.
- 128 people were killed and 222 people lost contact. Why was the earthquake damage in Japan’s Noto Peninsula so heavy?
- California suspect armed with ‘assault rifle,’ body armor killed after opening fire on deputies
- Tylertown Alicia Whatley Death Led To Car Accident: Obituary
- Family of 19-year-old found dead in suspicious circumstances near Alice Springs pleads for answers
- Tragic death: Layne Lemmel Car Accident, A Tragic Accident Claims the Life of Layne Lemmel, a Promising Bartender, In West Beaver Street
introduced or used in this paper
Results from the Paper
results from this paper
to get state-of-the-art GitHub badges and help the
community compare results to other papers.
No methods listed for this paper. Add
relevant methods here