Background. The swift advancement of Internet of Things (IoT) technology has revolutionized smart home settings; the prevalent automation systems are limited by their need for specific device identification and rigid rule-based configurations. These constraints impede natural human-device interaction, especially in dynamic or communal environments where spatial context is more instinctive than predetermined naming conventions. Current solutions frequently neglect spatial reasoning and multimodal inputs, resulting in heightened cognitive demands and diminished accessibility. The proposed work develops a spatial context-aware control system aimed at facilitating intuitive, vision-driven, and language-based interaction with smart devices to overcome these problems.

Methods. The proposed model modular, multimodal framework that integrates computer vision, natural language processing, and spatial inference for context-aware smart device control. The system comprises six core components: (i) an Onboarding Inference Engine for extracting device information via natural language input, (ii) Zero-Shot Device Detection using OWL-ViT for object identification without prior training, (iii) Metadata Refinement and Filtering for structured annotation and disambiguation, (iv) a Geospatial Device Visualizer for annotated visual feedback, (v) Spatial Topology Inference using GPT-4o for reasoning about physical layouts, and (vi) Intent-Based Command Synthesis with Gemini Flash to generate precise, executable control commands. The final Agentic Execution Module interfaces with the Tuya Smart Device API, ensuring vendor-agnostic actuation. The system supports multilingual input and adapts to various environmental contexts, including smart homes and assisted living facilities.

Results. A user study involving 15 participants (aged 18–80, diverse educational backgrounds) was conducted to evaluate the effectiveness of the proposed method in comparison to the Google Home Assistant. Quantitative findings demonstrate a statistically significant reduction in cognitive workload, with NASA Task Load Index (TLX) scores decreasing by an average of 13.17 points (p = 0.0013, Cohen’s d = 1.0381). Participants rated the proposed method higher in terms of ease of use (mean = 4.67) compared to Google Home (mean = 3.8) on a 5-point Likert scale. Qualitative feedback highlighted the intuitive nature of spatial context commands, reduced cognitive burden due to the elimination of device name memorization, and enhanced accessibility via support for regional languages. 93.3% of users preferred the proposed method over the baseline system. These results affirm the feasibility and user-centric benefits of integrating vision-language models for context-aware smart device control.