Muhammad Monjurul Karim
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Muhammad Monjurul Karim has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
- Ruwen Qin
Stony Brook University
6 shared publications Yinhai Wang
Department of Civil and Environmental Engineering, University of Washington, Seattle, WA
2 shared publications- Yu Li
Department of Civil Engineering, Stony Brook University, Stony Brook, NY, USA
2 shared publications - Zhaozheng Yin
Stony Brook University
2 shared publications - Bingzhang Wang
Department of Civil and Environmental Engineering, University of Washington, Seattle, WA
1 shared publication Chenyu Zhang
Department of Civil Engineering, Stony Brook University, Stony Brook, NY
1 shared publication
Latest Publications
- 6 May 2025
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
Thermal Forecast: Traffic Trajectory Prediction in Challenging Nighttime Conditions Using Thermal Imaging
Accurate trajectory prediction of traffic agents is crucial for the advancement of intelligent transportation systems (ITS), especially under challenging conditions such as nighttime and adverse weather. Although significant advances have been made in trajectory prediction for ITS, existing methods often fail in low-visibility environments because of their reliance on traditional red-green-blue cameras. Such cameras face significant limitations in low-light environments because of high glare, reflections, and reduced visibility. To address these challenges, this study presents a novel deep learning framework that leverages thermal cameras for robust trajectory prediction. We propose a vision transformer–based dual gated recurrent unit network for future trajectory prediction using thermal videos. The proposed network efficiently captures the complex interactions among object-specific and scene-level rich information in the temporal sequence from very limited cues in thermal images. Moreover, a custom loss function is proposed, which is designed to emphasize long-term prediction accuracy by penalizing errors more heavily in distant future predictions. The proposed method is evaluated in a challenging thermal video dataset, which is captured on rainy nights in different traffic intersections. Experimental results demonstrate promising performance across various prediction horizons for trajectory prediction of traffic agents in challenging nighttime conditions.
- 6 May 2025
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
Thermal Forecast: Traffic Trajectory Prediction in Challenging Nighttime Conditions Using Thermal Imaging
Accurate trajectory prediction of traffic agents is crucial for the advancement of intelligent transportation systems (ITS), especially under challenging conditions such as nighttime and adverse weather. Although significant advances have been made in trajectory prediction for ITS, existing methods often fail in low-visibility environments because of their reliance on traditional red-green-blue cameras. Such cameras face significant limitations in low-light environments because of high glare, reflections, and reduced visibility. To address these challenges, this study presents a novel deep learning framework that leverages thermal cameras for robust trajectory prediction. We propose a vision transformer–based dual gated recurrent unit network for future trajectory prediction using thermal videos. The proposed network efficiently captures the complex interactions among object-specific and scene-level rich information in the temporal sequence from very limited cues in thermal images. Moreover, a custom loss function is proposed, which is designed to emphasize long-term prediction accuracy by penalizing errors more heavily in distant future predictions. The proposed method is evaluated in a challenging thermal video dataset, which is captured on rainy nights in different traffic intersections. Experimental results demonstrate promising performance across various prediction horizons for trajectory prediction of traffic agents in challenging nighttime conditions.
- 28 February 2024
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
- Vol. 2678 (9) , 699-709
Fusion-GRU: A Deep Learning Model for Future Bounding Box Prediction of Traffic Agents in Risky Driving Videos
To ensure the safe and efficient navigation of autonomous vehicles and advanced driving assistance systems in complex traffic scenarios, predicting the future bounding boxes of surrounding traffic agents is crucial. However, simultaneously predicting the future location and scale of target traffic agents from the egocentric view poses challenges because the vehicles’ egomotion causes considerable field-of-view changes. Moreover, in anomalous or risky situations, tracking loss or abrupt motion changes limit the available observation time, requiring learning of cues within a short time window. Existing methods typically use a simple concatenation operation to combine different cues, overlooking their dynamics over time. To address this, this paper introduces the Fusion-Gated Recurrent Unit (Fusion-GRU) network, a novel encoder-decoder architecture for future bounding box localization. Unlike traditional gated recurrent units (GRUs), Fusion-GRU accounts for mutual and complex interactions among input features. Moreover, an intermediary estimator coupled with a self-attention aggregation layer is also introduced to learn sequential dependencies for long-range prediction. Finally, a GRU decoder is employed to predict the future bounding boxes. The proposed method is evaluated on two publicly available datasets, Risky Object Localization and Honda Egocentric Vehicle Intersection. The experimental results showcase the promising performance of the Fusion-GRU, demonstrating its effectiveness in predicting future bounding boxes of traffic agents.
- 28 February 2024
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
- Vol. 2678 (9) , 699-709
Fusion-GRU: A Deep Learning Model for Future Bounding Box Prediction of Traffic Agents in Risky Driving Videos
To ensure the safe and efficient navigation of autonomous vehicles and advanced driving assistance systems in complex traffic scenarios, predicting the future bounding boxes of surrounding traffic agents is crucial. However, simultaneously predicting the future location and scale of target traffic agents from the egocentric view poses challenges because the vehicles’ egomotion causes considerable field-of-view changes. Moreover, in anomalous or risky situations, tracking loss or abrupt motion changes limit the available observation time, requiring learning of cues within a short time window. Existing methods typically use a simple concatenation operation to combine different cues, overlooking their dynamics over time. To address this, this paper introduces the Fusion-Gated Recurrent Unit (Fusion-GRU) network, a novel encoder-decoder architecture for future bounding box localization. Unlike traditional gated recurrent units (GRUs), Fusion-GRU accounts for mutual and complex interactions among input features. Moreover, an intermediary estimator coupled with a self-attention aggregation layer is also introduced to learn sequential dependencies for long-range prediction. Finally, a GRU decoder is employed to predict the future bounding boxes. The proposed method is evaluated on two publicly available datasets, Risky Object Localization and Honda Egocentric Vehicle Intersection. The experimental results showcase the promising performance of the Fusion-GRU, demonstrating its effectiveness in predicting future bounding boxes of traffic agents.
- 11 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (1) , 1792-1803
An Attention-Guided Multistream Feature Fusion Network for Early Localization of Risky Traffic Agents in Driving Videos
Detecting dangerous traffic agents in videos captured by vehicle-mounted dashboard cameras (dashcams) is essential to ensure safe navigation in complex environments. Accident-related videos are just a minor portion of the driving-related big data, and the transient pre-accident process is highly dynamic and complex. Besides, risky and non-risky traffic agents can be similar in their appearance. These make risky traffic agent localization in the driving video particularly challenging. To this end, this paper proposes an attention-guided multistream feature fusion network (AM-Net) to localize dangerous traffic agents from dashcam videos ahead of potential accidents. Two Gated Recurrent Unit (GRU) networks use object bounding box and optical flow features extracted from consecutive video frames to capture spatio-temporal cues for distinguishing risky traffic agents. An attention module, coupled with the GRUs, learns to identify traffic agents that are relevant to an accident. Fusing the two streams of global and object-level features, AM-Net predicts the riskiness scores of traffic agents in the video. In supporting this study, the paper also introduces a new benchmark dataset called Risky Object Localization (ROL). The dataset contains spatial, temporal, and categorical annotations of the accident, object, and scene-level attributes. The proposed AM-Net achieves a promising performance of 85.59% AUC on the ROL dataset. Additionally, the AM-Net outperforms the current state-of-the-art for video anomaly detection by 3.5% AUC on the public DoTA dataset. A thorough ablation study further reveals AM-Net's merits by assessing the impact of its constituents.
- 11 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (1) , 1792-1803
An Attention-Guided Multistream Feature Fusion Network for Early Localization of Risky Traffic Agents in Driving Videos
Detecting dangerous traffic agents in videos captured by vehicle-mounted dashboard cameras (dashcams) is essential to ensure safe navigation in complex environments. Accident-related videos are just a minor portion of the driving-related big data, and the transient pre-accident process is highly dynamic and complex. Besides, risky and non-risky traffic agents can be similar in their appearance. These make risky traffic agent localization in the driving video particularly challenging. To this end, this paper proposes an attention-guided multistream feature fusion network (AM-Net) to localize dangerous traffic agents from dashcam videos ahead of potential accidents. Two Gated Recurrent Unit (GRU) networks use object bounding box and optical flow features extracted from consecutive video frames to capture spatio-temporal cues for distinguishing risky traffic agents. An attention module, coupled with the GRUs, learns to identify traffic agents that are relevant to an accident. Fusing the two streams of global and object-level features, AM-Net predicts the riskiness scores of traffic agents in the video. In supporting this study, the paper also introduces a new benchmark dataset called Risky Object Localization (ROL). The dataset contains spatial, temporal, and categorical annotations of the accident, object, and scene-level attributes. The proposed AM-Net achieves a promising performance of 85.59% AUC on the ROL dataset. Additionally, the AM-Net outperforms the current state-of-the-art for video anomaly detection by 3.5% AUC on the public DoTA dataset. A thorough ablation study further reveals AM-Net's merits by assessing the impact of its constituents.
- 28 February 2023
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
- Vol. 2677 (7) , 693-704
A Multitask Deep Learning Model for Parsing Bridge Elements and Segmenting Defect in Bridge Inspection Images
The vast network of bridges in the United States has a high requirement for maintenance and rehabilitation. The massive cost of manual visual inspection to assess bridge conditions is a burden to some extent. Advanced robots have been used to automate inspection data collection. Automating the segmentations of multiclass elements and surface defects on the elements in the large volume of inspection image data would facilitate an efficient and effective assessment of bridge conditions. Training separate single-task networks for element parsing (i.e., semantic segmentation of multiclass elements) and defect segmentation fails to incorporate the close connection between these two tasks. Both recognizable structural elements and apparent surface defects are present in the inspection images. This paper is motivated to develop a multitask deep learning model that fully uses such interdependence between bridge elements and defects to boost the model’s task performance and generalization. Furthermore, the study investigated the effectiveness of the proposed model designs for improving task performance, including feature decomposition, cross-talk sharing, and multi-objective loss function. A data set with pixel-level labels of bridge elements and corrosion was developed for model training and testing. Quantitative and qualitative results from evaluating the developed multitask deep model demonstrate its advantages over the single-task-based model not only in performance (2.59% higher mIoU on bridge parsing and 1.65% on corrosion segmentation) but also in computational time and implementation capability.
- 28 February 2023
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
- Vol. 2677 (7) , 693-704
A Multitask Deep Learning Model for Parsing Bridge Elements and Segmenting Defect in Bridge Inspection Images
The vast network of bridges in the United States has a high requirement for maintenance and rehabilitation. The massive cost of manual visual inspection to assess bridge conditions is a burden to some extent. Advanced robots have been used to automate inspection data collection. Automating the segmentations of multiclass elements and surface defects on the elements in the large volume of inspection image data would facilitate an efficient and effective assessment of bridge conditions. Training separate single-task networks for element parsing (i.e., semantic segmentation of multiclass elements) and defect segmentation fails to incorporate the close connection between these two tasks. Both recognizable structural elements and apparent surface defects are present in the inspection images. This paper is motivated to develop a multitask deep learning model that fully uses such interdependence between bridge elements and defects to boost the model’s task performance and generalization. Furthermore, the study investigated the effectiveness of the proposed model designs for improving task performance, including feature decomposition, cross-talk sharing, and multi-objective loss function. A data set with pixel-level labels of bridge elements and corrosion was developed for model training and testing. Quantitative and qualitative results from evaluating the developed multitask deep model demonstrate its advantages over the single-task-based model not only in performance (2.59% higher mIoU on bridge parsing and 1.65% on corrosion segmentation) but also in computational time and implementation capability.
- 25 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Human-Machine Systems
- Vol. 52 (4) , 591-601
A Virtual-Reality-Based Training and Assessment System for Bridge Inspectors With an Assistant Drone
Over 600 000 bridges in the U.S. must be inspected every two years to identify flaws, defects, or potential problems that may need follow-up maintenance. Bridge inspection has adopted unmanned aerial vehicles (or drones) for improving safety, efficiency, and cost-effectiveness. Although drones can operate in an autonomous mode, keeping inspectors in the loop is critical for complex tasks in bridge inspection. Therefore, inspectors need to develop the skill and confidence to operate drones in their jobs. This article presents the design and development of a virtual-reality-based training and assessment system for inspectors assisted by a drone in bridge inspection. The system is composed of four integrated modules: a simulated bridge inspection developed in Unity, an interface that allows a trainee to operate the drone in simulation using a remote controller, data monitoring and analysis to provide real-time in-task feedback to trainees to assist their learning, and a post-study assessment supporting personalized training. This article also conducts a proof-of-concept pilot study to illustrate the functionality of this system. The study demonstrated that the training and assessment system for bridge inspection with an assistant drone, as a tool for the early-stage training, can objectively identify the training needs of individuals in detail and, further, help them develop the skill and confidence in collaborating with a drone in bridge inspection. The system has built a modeling and analysis platform for exploring advanced solutions to the human–drone cooperative inspection of civil infrastructure.
- 25 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Human-Machine Systems
- Vol. 52 (4) , 591-601
A Virtual-Reality-Based Training and Assessment System for Bridge Inspectors With an Assistant Drone
Over 600 000 bridges in the U.S. must be inspected every two years to identify flaws, defects, or potential problems that may need follow-up maintenance. Bridge inspection has adopted unmanned aerial vehicles (or drones) for improving safety, efficiency, and cost-effectiveness. Although drones can operate in an autonomous mode, keeping inspectors in the loop is critical for complex tasks in bridge inspection. Therefore, inspectors need to develop the skill and confidence to operate drones in their jobs. This article presents the design and development of a virtual-reality-based training and assessment system for inspectors assisted by a drone in bridge inspection. The system is composed of four integrated modules: a simulated bridge inspection developed in Unity, an interface that allows a trainee to operate the drone in simulation using a remote controller, data monitoring and analysis to provide real-time in-task feedback to trainees to assist their learning, and a post-study assessment supporting personalized training. This article also conducts a proof-of-concept pilot study to illustrate the functionality of this system. The study demonstrated that the training and assessment system for bridge inspection with an assistant drone, as a tool for the early-stage training, can objectively identify the training needs of individuals in detail and, further, help them develop the skill and confidence in collaborating with a drone in bridge inspection. The system has built a modeling and analysis platform for exploring advanced solutions to the human–drone cooperative inspection of civil infrastructure.
- Top Cited Papers
- 9 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 23 (7) , 9590-9600
A Dynamic Spatial-Temporal Attention Network for Early Anticipation of Traffic Accidents
The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipation of traffic accidents remains a challenge. Inspired by the attention behavior of humans in visually perceiving accident risks, this paper proposes a Dynamic Spatial-Temporal Attention (DSTA) network for the early accident anticipation from dashcam videos. The DSTA-network learns to select discriminative temporal segments of a video sequence with a Dynamic Temporal Attention (DTA) module. It also learns to focus on the informative spatial regions of frames with a Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is trained jointly with the attention modules to predict the probability of a future accident. The evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study that assesses the DSTA-network at the component level reveals how the network achieves such performance. Furthermore, this paper proposes a method to fuse the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.
- 9 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 23 (7) , 9590-9600
Top Cited PapersA Dynamic Spatial-Temporal Attention Network for Early Anticipation of Traffic Accidents
The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipation of traffic accidents remains a challenge. Inspired by the attention behavior of humans in visually perceiving accident risks, this paper proposes a Dynamic Spatial-Temporal Attention (DSTA) network for the early accident anticipation from dashcam videos. The DSTA-network learns to select discriminative temporal segments of a video sequence with a Dynamic Temporal Attention (DTA) module. It also learns to focus on the informative spatial regions of frames with a Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is trained jointly with the attention modules to predict the probability of a future accident. The evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study that assesses the DSTA-network at the component level reveals how the network achieves such performance. Furthermore, this paper proposes a method to fuse the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.
- 18 February 2022
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
- Vol. 2676 (6) , 743-755
Toward Explainable Artificial Intelligence for Early Anticipation of Traffic Accidents
Traffic accident anticipation is a vital function of Automated Driving Systems (ADS) in providing a safety-guaranteed driving experience. An accident anticipation model aims to predict accidents promptly and accurately before they occur. Existing Artificial Intelligence (AI) models of accident anticipation lack a human-interpretable explanation of their decision making. Although these models perform well, they remain a black-box to the ADS users who find it to difficult to trust them. To this end, this paper presents a gated recurrent unit (GRU) network that learns spatio-temporal relational features for the early anticipation of traffic accidents from dashcam video data. A post-hoc attention mechanism named Grad-CAM (Gradient-weighted Class Activation Map) is integrated into the network to generate saliency maps as the visual explanation of the accident anticipation decision. An eye tracker captures human eye fixation points for generating human attention maps. The explainability of network-generated saliency maps is evaluated in comparison to human attention maps. Qualitative and quantitative results on a public crash data set confirm that the proposed explainable network can anticipate an accident on average 4.57 s before it occurs, with 94.02% average precision. Various post-hoc attention-based XAI methods are then evaluated and compared. This confirms that the Grad-CAM chosen by this study can generate high-quality, human-interpretable saliency maps (with 1.23 Normalized Scanpath Saliency) for explaining the crash anticipation decision. Importantly, results confirm that the proposed AI model, with a human-inspired design, can outperform humans in accident anticipation.
- 18 February 2022
- Published by SAGE Publications in Transportation Research Record: Journal of the Transportation Research Board
- Vol. 2676 (6) , 743-755
Toward Explainable Artificial Intelligence for Early Anticipation of Traffic Accidents
Traffic accident anticipation is a vital function of Automated Driving Systems (ADS) in providing a safety-guaranteed driving experience. An accident anticipation model aims to predict accidents promptly and accurately before they occur. Existing Artificial Intelligence (AI) models of accident anticipation lack a human-interpretable explanation of their decision making. Although these models perform well, they remain a black-box to the ADS users who find it to difficult to trust them. To this end, this paper presents a gated recurrent unit (GRU) network that learns spatio-temporal relational features for the early anticipation of traffic accidents from dashcam video data. A post-hoc attention mechanism named Grad-CAM (Gradient-weighted Class Activation Map) is integrated into the network to generate saliency maps as the visual explanation of the accident anticipation decision. An eye tracker captures human eye fixation points for generating human attention maps. The explainability of network-generated saliency maps is evaluated in comparison to human attention maps. Qualitative and quantitative results on a public crash data set confirm that the proposed explainable network can anticipate an accident on average 4.57 s before it occurs, with 94.02% average precision. Various post-hoc attention-based XAI methods are then evaluated and compared. This confirms that the Grad-CAM chosen by this study can generate high-quality, human-interpretable saliency maps (with 1.23 Normalized Scanpath Saliency) for explaining the crash anticipation decision. Importantly, results confirm that the proposed AI model, with a human-inspired design, can outperform humans in accident anticipation.