IIIT-Hyderabad uses AI to decode the Indian thali

Researchers at IIIT–Hyderabad are developing an AI-powered system to analyse the Indian thali, a meal known for its diverse textures and overlapping ingredients. The research was recently presented at the 16th Indian Conference on Computer Vision, Graphics and Image Processing 2025.

Hyderabad: Can artificial intelligence understand the typical Indian thali comprising different food varieties, with mixed textures and overlapping ingredients? The International Institute of Information Technology (IIIT) Hyderabad researchers say it can.

Researchers from the Centre for Visual Information Technology at IIIT–Hyderabad, using computer vision and artificial intelligence, are developing tools to understand a typical Indian thali comprising multiple dishes such as rice, dal, roti, chutney, curd and papad.

---

The work is aimed at tracking calories in Indian thalis besides conserving food culture and traditions. Prof. CV Jawahar, who has been leading the project said, “If you are given a full plate of typical Indian food that not only has multiple dishes, but mixed ones like rice topped with dal, a roti hidden under a papad.… how do you understand what is there on a plate and eventually its nutritional value?”

This problem is critical as most existing food-tracking apps are designed for discrete and standardized Western meals. Existing food-scanner and nutrition systems assume a fixed menu and stable recipes. Indian food breaks both assumptions.

Instead of retraining models, the team built a zero-shot system, which can recognise new food items without starting over. According to researchers, the system first identifies food regions without knowing exactly what the food item is. Then, instead of rigid classification, the system uses retrieval-based prototype matching. This approach makes the system scalable, flexible, and realistic for places like cafeterias and hospital messes.

The research began not in a restaurant but with a healthcare need. The request was raised particularly around monitoring nutrition for pregnant women, and that helped shape this work.

Accordingly, the IIIT–Hyderabad research team built a zero-shot system. It first identifies regions on the plate and then matches them with visual examples, allowing it to work even when the menu changes overnight in cafeterias or hospital messes.

The current setup works with an overhead camera at a kiosk. “We want to extend it to an app-based system,” said Yash Arora, one of the researchers. To prepare for this, the team captured data from multiple angles, allowing food to be scanned from a phone camera – not just a fixed setup. The team, however, acknowledged that there are many practical issues still to be solved.

The research, which resulted in a paper titled ”What is there in an Indian thali” authored by Yash Arora and Aditya Arun under the guidance of Prof. Jawahar was presented at the 16th Indian Conference on Computer Vision, Graphics and Image Processing 2025.