Indoor anomaly detection in smart city environments

Abstract

Hotel corridors often have unattended items like cutlery and utensils left on the corridors, which is unappealing and a trip hazard. Going around the hotels and clearing these items is time-consuming and labour-intensive. Automating this process can reduce workload. This thesis presents a multi-stage machine vision system to detect utensils such as cups, plates, glasses, trays, and cutlery items, including spoons, forks, and butterknives. The system integrates YOLO for object detection with ConvNeXt and ViT as classification models. The system also incorporates the Segment Anything Model (SAM) for improved object isolation. Custom datasets were created for the training and testing of the system in various environments. Different systems were created by using the above-mentioned models, and rigorous testing was conducted to evaluate all the systems' robustness under diverse conditions using precision, recall, and F1-score metrics, with the best-performing system achieving an F1 score of 0.7161 in the corridor setting. The system was then integrated with a robot, enabling real-time object detection. This work contributes to a robust, adaptable system for automated monitoring in the hospitality sector.