MTKGR: multi-task knowledge graph reasoning for food and ingredient recognition

MTKGR: multi-task knowledge graph reasoning for food and ingredient recognition

연구 분야: Databases

학회: Multimedia Systems

Food and ingredient recognition emerges as a pivotal challenge in the domain of computer vision, particularly pertinent to multimedia systems applications. To exploit the intricate relationships between foods and their constituent ingredients, this paper introduces a novel approach termed Multi-Task Knowledge Graph Reasoning for Food and Ingredient Recognition (MTKGR). By integrating a multi-task convolutional neural network model with knowledge graph reasoning, MTKGR achieves significant breakthroughs in ingredient recognition on the ETH Food-101 and Ingredient-101 datasets, propelling the Micro-F1 and Macro-F1 scores to new state-of-the-art heights with improvements of 2.23% and 0.83%, respectively. Specifically, the multi-task model performs joint food and ingredient recognition, while the knowledge graph captures associations between food and ingredient entities. Knowledge graph reasoning is then applied to reconcile errors and inconsistencies in the multi-task model’s predictions. Our proposed Precision and Logits Ranking technique identifies the optimal food-ingredient label combination with maximum concordance with the knowledge graph. The substantial results not only demonstrate MTKGR’s potential in food and ingredient recognition but also showcase the value of fusing deep learning and symbolic reasoning for enhanced visual understanding and intelligent analysis within multimedia systems.