Monday, Oct 2, 2023

IIITH part of Facebook AI’s pathbreaking Ego4D

Hyderabad: That nagging doubt of whether you switched off the iron box or the gas stove, or whether you locked the car much after you left home or the car in the parking lot is all too familiar for many. But imagine Artificial Intelligence-driven devices like augmented reality (AR) glasses telling you that you did […]

By Dennis Marcus Mathew

Updated On - 12:16 AM, Tue - 19 October 21

IIITH part of Facebook AI’s pathbreaking Ego4D

Hyderabad: That nagging doubt of whether you switched off the iron box or the gas stove, or whether you locked the car much after you left home or the car in the parking lot is all too familiar for many. But imagine Artificial Intelligence-driven devices like augmented reality (AR) glasses telling you that you did lock the car, or helping you find your lost keys.

This could soon be a reality, courtesy Ego4D, a project initiated by Facebook AI in collaboration with 13 partner institutes and labs from countries including the UK, Italy, India, Japan, Singapore and the United States. This November, they will unveil a mammoth dataset comprising over 2,200 hours of first person videos of over 700 participants engaged in routine activities.

And Hyderabadis can hold their heads up high, with the city’s premier International Institute of Information Technology (IIIT) Hyderabad being the only Indian institute in the global consortium for this project that promises to enable experiences straight out of sci-fi.

What makes the Ego4D project novel is the manner in which data was collected. To enable pathbreaking research in immersive experiences, the dataset comprises video footage from a first person’s perspective, known among scientists as ‘egocentric perception’.

According to Kristen Grauman, lead research scientist at Facebook AI, these are “videos that show the world from the centre of action, rather than the sidelines”. The footage has been collected via head-mounted devices combined with other egocentric sensors. By recognising the location, scene of activity and social relationships, these sort of devices could be trained to not only automatically understand what the wearer is looking at, attending to, or even manipulating, but also the context of the social situation itself, according to an IIITH blog.

IIITH’s role

The IIITH collected data from over 130 participants spread across 25 locations in the country. According to Prof CV Jawahar, Centre for Visual Information Technology, IIITH, the pandemic forced the institute to find multiple local teams and ship cameras as well as data and then train people over videos.

At each location, participants spanned a gamut of vocations and activities from home cooks to carpenters, painters, electricians and farmers.

“A wearable device with first person vision can help someone who may have some visual impairment,” says Prof Jawahar. A similar sort of ‘assistance’ can help in reinforcing memory especially for those exhibiting early signs of dementia or memory disorders. “Even while conducting surgeries, it can guide and provide additional cues to the surgeon wearing the device,” he adds.