Newsroom

Sudeep Sarkar Awarded Collaborative National Science Foundation (NSF) Grant to build computer vision/AI algorithms that understand events from videos

June 23, 2020

Sarkar

CSE Professor Sudeep Sarkar is the lead of a collaborative NSF grant for the project “Understanding Events from Streaming Video – Joint Deep and Graph Representations, Commonsense Priors, and Predictive Learning.” Of the $1,005,543 grant, $421,180 was awarded to USF. Fellow collaborators are Anuj Srivastava from Florida State University and Sathya Aakur from Oklahoma State University.

While it is easy for humans to process video data and extract meanings from it, it is extremely hard to design algorithms to do so. When developed, there are many applications of this technology, such as building assistive robotics or constructing smart spaces for independent living or monitoring wildlife.  Video-data capture events, which are central to the content of human experience. Events consist of objects/people (who), location (where), time (when), actions (what), activities (how), and intent (why). The goal of this proposed study is to formulate a computer vision-based event understanding algorithm that operates in a self-supervised, streaming fashion. The algorithm will predict and detect old and new events, learn to build hierarchical event representations, all in the context of a prior knowledge-base that is updated over time. The intent is to generate interpretations of an event that go beyond what is seen, rather than just recognition.  This research pushes the frontier of computer vision by coupling the self-supervised learning process with prior knowledge, moving the field towards open-world algorithms, and needing little or no supervision. Furthermore, this project will focus on recruitment and retention of undergraduate women students through freshman and sophomore years, with attention towards students from populations underrepresented in computing at the three sites: University of South Florida, Florida State University, and Oklahoma State University.