CECI Develops Site Image Summarization System Linking Multimodal AI and Safety Regulations

Analyzing massive daily construction site photos manually is time-consuming and often inconsistent. Addressing these operational challenges, CECI Engineering Consultants collaborated with Tamkang University and National Yunlin University of Science and Technology to introduce an AI construction site image text summarization system. The framework combines multimodal vision models, LLMs for reasoning and summary generation, and retrieval technology linked to structured safety regulations. Field tests demonstrated high accuracy in identifying specific machinery defects, though unstructured issues like environmental sanitation presented ongoing challenges.

Key takeaways

• The system combines multimodal models, LLMs, and image retrieval to extract visual features, reason about hazard causes, and query structured regulatory databases to reduce citation errors.
• Expert evaluations across multiple safety categories showed average scores between 4.00 and 4.60, with heavy machinery and improper equipment use scoring highest at 4.60.
• Environmental sanitation scored lower in defect description accuracy (2.40), mainly due to limited training samples and difficulty in quantifying unstructured clutter or standing water.
• Future milestones include expanding regulation databases, enhancing visual comprehension, refining retrieval precision, and integrating with internal PMIS platforms.