Computer vision-based deep learning for supervising excavator operations and measuring real-time earthwork productivity

Min Yuan Cheng, Minh Tu Cao*, Christian Kentaro Nuralim

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

7 Scopus citations


Its significant influence on overall progress makes excavator productivity a major concern of construction project managers, particularly during initial-stage earthwork activities. The ability to track excavator performance autonomously in real time is essential to monitoring and managing earthwork effectively. A novel, vision-based autonomous excavator action recognition and productivity measurement is introduced in this study that integrates the you only watch once deep learning method to recognize excavator actions, which are used to calculate action time and average cycle time. The average cycle time is then used to calculate the productivity of excavation work. The images of the excavator action used to construct the model were extracted from a video recorded at the construction site. The algorithm recognized excavator actions with an F1 score of 87.6% and mAP value of 81.6%. The outputs were then used in the proposed framework to measure excavator productivity. Omitting the outliers/misclassifications (< 95% confidence) before calculating average action times was found to yield a cycle time accuracy (99.7%) that was significantly higher than that achieved using the unadjusted mean value (81.59%). The successful implementation of the proposed framework in this study demonstrates the feasibility of using the proposed autonomous productivity measurement in construction as a more economical, faster, and real-time measure of excavator productivity.

Original languageEnglish
Pages (from-to)4468-4492
Number of pages25
JournalJournal of Supercomputing
Issue number4
StatePublished - Mar 2023


  • Action recognition
  • Cycle time
  • Deep learning
  • Excavation productivity
  • You only watch once


Dive into the research topics of 'Computer vision-based deep learning for supervising excavator operations and measuring real-time earthwork productivity'. Together they form a unique fingerprint.

Cite this