SENSE.nano 2021: Multimodal representation learning via maximization of local mutual information

Ruizhi (Ray) Liao, Postdoctoral Associate, MIT Computer Science & Artificial Intelligence Lab Abstract: Liao proposes and demonstrates a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Liao's method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. He makes use of recent advances in mutual information estimation with neural network discriminators. Liao argues that the sum of local mutual information is typically a lower bound on the global mutual information. His experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Biography: Liao earned his computer science PhD at MIT in Aug 2021, advised by Prof Polina Golland. He studies machine learning and develop computational tools driven by clinical problems. Liao is excited about ubiquitous computing and its potential to advance health care. His PhD research has been supported by Merrill Lynch Fellowship and Siebel Fellowship. ---------------------------------------------------- The 2021 SENSE.nano symposium focused on human subjects research, exploring how sensors and sensing systems can enable current medical studies and future clinical practice. SENSE.nano 2021 also celebrated the re-opening of the expanded Clinical Research Center (CRC) at MIT, now known as MIT's Center for Clinical and Translational Research. Broken into two half-day webinars, SENSE.nano 2021 investigated human health through various technologies including motion capture, physiological monitoring, and sensing tools for the study of bodily fluids. Over a series of invited technical talks, panel discussions, presentations by MIT-launched startups, and views into MIT research today with current graduate students, this event provided needs context and solution perspectives in the domains of sensing for medical engineering and science, and for the care of humans in their environment. The 2021 SENSE.nano Symposium was sponsored by MIT.nano, MIT's Center for Clinical and Translational Research, and the MIT Industrial Liaison Program (ILP).