AI-assisted Multi-Modal Automotive Diagnostic System for Fault Identification and Localization
Date Issued
May 2026
Author(s)
Al Zaraa, Ali
Advisor
Abstract
Experienced mechanics listen for unusual sounds and feel for vibrations when diagnosing a fault but standard onboard diagnostic systems ignore these cues entirely. This thesis describes the design, build, and testing of an AI-assisted diagnostic system that fills this gap by combining standard vehicle data with externally mounted acoustic and vibration sensors.
The system is built around two repositionable ESP32-based sensor nodes, each integrating an INMP441 MEMS microphone and an ADXL345 three-axis accelerometer, which are magnetically mounted on the vehicle during a diagnostic session. A third ESP32 acts as a wireless bridge, forwarding data to a central Raspberry Pi 4 Model B that coordinates acquisition, AI inference, and visualization through a touchscreen dashboard. Live engine parameters are retrieved from the vehicle via an OBD-II interface, and an MLX90640 thermal infrared camera provides real-time heat mapping of the engine bay as a supplementary visual aid.
The artificial intelligence component is structured as a two-stage pipeline. The first stage uses location-specific TensorFlow Lite models, each built on a MobileNetV2 backbone, to classify two-second audio clips into fault categories from log-mel spectrograms. The second stage is a transparent rule-based fusion layer that combines the audio prediction with live engine RPM and vibration statistics through three steps: eliminating fault classes that are physically implausible at the current operating point, ranking the survivors with a weighted score, and reporting a final confidence value the technician can interpret. The system covers nine fault categories across three diagnostic zones: engine front, front wheel, and rear wheel.
A dataset of 605 labelled audio clips was collected and the architecture was refined through three rounds of experimentation, reaching an overall classification accuracy of 93.55%. The system supports continuous improvement through an incremental learning workflow in which technicians can correct predictions and add verified clips to the dataset for subsequent retraining.
The results show that combining acoustic, vibration, and OBD data in a location-aware framework clearly outperforms using any single data source alone. Expanding the sensor network, adding more OBD parameters, and tuning per vehicle type are the most direct routes to improving it further
The system is built around two repositionable ESP32-based sensor nodes, each integrating an INMP441 MEMS microphone and an ADXL345 three-axis accelerometer, which are magnetically mounted on the vehicle during a diagnostic session. A third ESP32 acts as a wireless bridge, forwarding data to a central Raspberry Pi 4 Model B that coordinates acquisition, AI inference, and visualization through a touchscreen dashboard. Live engine parameters are retrieved from the vehicle via an OBD-II interface, and an MLX90640 thermal infrared camera provides real-time heat mapping of the engine bay as a supplementary visual aid.
The artificial intelligence component is structured as a two-stage pipeline. The first stage uses location-specific TensorFlow Lite models, each built on a MobileNetV2 backbone, to classify two-second audio clips into fault categories from log-mel spectrograms. The second stage is a transparent rule-based fusion layer that combines the audio prediction with live engine RPM and vibration statistics through three steps: eliminating fault classes that are physically implausible at the current operating point, ranking the survivors with a weighted score, and reporting a final confidence value the technician can interpret. The system covers nine fault categories across three diagnostic zones: engine front, front wheel, and rear wheel.
A dataset of 605 labelled audio clips was collected and the architecture was refined through three rounds of experimentation, reaching an overall classification accuracy of 93.55%. The system supports continuous improvement through an incremental learning workflow in which technicians can correct predictions and add verified clips to the dataset for subsequent retraining.
The results show that combining acoustic, vibration, and OBD data in a location-aware framework clearly outperforms using any single data source alone. Expanding the sensor network, adding more OBD parameters, and tuning per vehicle type are the most direct routes to improving it further
Affiliation
File(s)![Thumbnail Image]()
Name
Ali Al Zaraa.bsc.2026.abstract.pdf
Size
114.05 KB
Format
Adobe PDF
Checksum (MD5)
ec798d1bd8b31c493a92188df73f557c