I am a PhD student at New York University's Center for Data Science advised by Prof. Yann LeCun and Prof. Kyunghyun Cho, where I work on computer vision and natural language processing. Prior to this I was advised during my Masters by Prof. Andrew McCallum at University of Massachusetts Amherst in areas of natural language processing with a special focus on structured prediction.

My current interests lie at the intersection of vision and language, and my research focuses on using information from multiple sources such as text, images, and video to improve commonsense reasoning capabilities of machines. During my PhD, I hope to build multi-modal methods that are robust, reliable and interpretable and can be used in assistive technology.

If you are a NYU Master's student looking to work on research, especially related to multi-modal learning, feel free to reach out to me! :)

Email  /  CV  /  Google Scholar  /  Twitter  /  Github

profile photo



MDETR - Modulated Detection for End-to-End Multi-modal Understanding
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, Nicolas Carion
ICCV 2021,   (Oral Presentation, top 3% of submissions)
Project page / Paper / Code & Model weights / Colab

We step away from existing approaches to multi-modal understanding that involve frozen pre-trained object detectors trained on a fixed label set, and instead achieve true end-to-end multi-modal understanding by detecting objects that are referred to in free form text. You can now detect and reason over novel combination of object classes and attributes like "a pink elephant"!

AdapterFusion: Non-Destructive Task Composition for Transfer Learning
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych
EACL 2021,   (Oral Presentation)
Project page & Code / Paper / Colab

We propose a new transfer learning algorithm that combines skills learned from multiple tasks in a non-destructive manner.

A Survey on Semantic Parsing
Aishwarya Kamath, Rajarshi Das
AKBC 2019

A brief history of semantic parsing, with pointers to several seminal works. Check out the poster for a TL;DR.

Specializing Distributional Vectors of All Words for Lexical Entailment
Aishwarya Kamath*, Jonas Pfeiffer*, Edoardo M. Ponti, Goran Glavaš, , Ivan Vulic´
Representation Learning for NLP Workshop, ACL 2019,   (Best Paper Award)

We present the first word embedding post-processing method that specializes vectors of all vocabulary words – including those unseen in the resources – for the asymmetric relation of lexical entailment (LE) (i.e., hyponymy-hypernymy relation). We report consistent gains over state-of-the-art LE-specialization methods, and successfully LE-specialize word vectors for languages without any external lexical knowledge.

Training Structured Prediction Energy Networks with Indirect Supervision
Amirmohammad Rooshenas, Aishwarya Kamath, Andrew McCallum,
NAACL 2018,   (Oral Presentation)

We train a structured prediction energy network (SPEN) without any labeled data instances, where the only source of supervision is a simple human-written scoring function.


Academic Service