Our framework comprises key video clips extraction, trade recognition and worker competency evaluation. Trade recognition is a new proposed method through analyzing the dynamic spatiotemporal relevance between workers and non-worker objects. We also improved the identification results by analyzing, comparing, and matching multiple face images of each worker obtained from videos. The experimental results demonstrate the reliability and accuracy of our deep learning-based method to detect workers who are carrying out work for which they are not certified to facilitate safety inspection and supervision.