To prevent power outages, electric utilities regularly perform visual inspections on their power grids to plan for necessary repair or replacement work. These inspections have typically been carried out using traditional methods such as foot patrol and helicopter-assisted surveys, which are typically slow, expensive, and potentially dangerous. To improve inspection speed, accuracy, and to reduce inspection costs, a considerable amount of research has been conducted in order to automate vision-based power line inspection.

In a paper published in IEEE Power and Energy Technology Systems Journal, we present a novel automatic autonomous vision-based power line inspection system that uses unmanned aerial vehicle inspection as the main inspection method. We used optical images as the primary data source, and deep learning as the backbone of the data analysis.

Our research demonstrates the potential role and the importance of automatic autonomous power line inspection in the intelligent monitoring of power grids.

Three main challenges

We address three main challenges of deep learning in vision-based power line inspection:

1. The lack of training data
2. Class imbalance
3. The detection of small components and defects

Having addressed the three challenges, we built a basic automatic vision-based power line inspection system with two custom-built UAVs and five deep learning-based models for data analysis and inspection.

First, we created four medium-sized datasets for training component detection and classification models. Next, we proposed a multi-stage component detection and classification pipeline to detect small power components and defects. The pipeline consists of five components: a mast detector, a component detector and three component classifiers. The pipeline works as follows: First, the mast detector detects power masts from input images. Then, the detected masts are cropped from the input images and used as inputs for the component detector to detect power components including top caps, poles, cross arms, and insulators. The detected top caps, poles, and cross arms are cropped from the input images and passed through their corresponding classifiers to identify defects.

Finally, we proposed a series of effective data augmentation techniques to generate more training data and balance out the imbalanced classes. To train a robust component detector for our pipeline, we combined original images with mast crops generated by the mast detector. When a mast is detected, its predicted bounding box is padded to be a square and cropped from the original image to generate one additional training image. The square is also slightly shifted in four directions (left, right, top, bottom) to generate four more training images. In addition, the training images are randomly flipped during training. These data augmentation techniques allow us to generate a training set that is 12 times bigger than our original training set. 

The result indicates that the proposed approaches can address the three challenges and deliver significant improvement for detecting and classifying power line components.

Read also: Better utilize your data with AI 

Potential roles in the intelligent monitoring of power grids

By combining these technologies the current version of eSmart Systems’ software is capable of detecting common power line components, such as poles, cross arms, top caps, insulators, and inspecting common defects on power line components including missing top caps, cracked poles, woodpecker-damaged poles, cracked cross arms, and rot-damaged cross arms. When deployed in the Microsoft Azure cloud, with autoscale functionality and access to GPU VMs, our system has demonstrated its ability to analyze over 180,000 images per hour. This allows power utilities to inspect their power grids more often and at a lower cost than traditional inspection methods.

Our system gives energy companies a fast and efficient tool to view the status of their infrastructure as well as export reports as a basis for their maintenance tasks. In addition, with the ability to access hard-to-reach areas and fly at high speed, our UAVs allow almost immediate assessment of power line damage after natural disasters for energy companies to plan for immediate repair or replacement work, which can greatly reduce the outage time and quickly reconnect the power grid.

Our system has demonstrated its potential roles in the intelligent monitoring of power grids. Autonomous UAVs equipped with sensors and cameras can automatically fly along power lines to perform online brief inspection to identify serious defects (e.g., collapsed poles, broken power lines, trees lying across and against power lines) and collect data for offline thorough inspection to detect potential defects that may lead to power outages such as broken insulators, missing top caps, cracked poles, woodpecker damaged poles, cracked cross arms, and rot-damaged cross arms. These potential defects are very important information sources for electric utilities to make decisions for necessary repair or replacement work before any major damage that may cause power blackout.

Next steps

Two potential next steps could be interesting to follow-up. The first step is to combine the GPS way points-based, pole detection-based, and power line detection-based navigation approaches with UAV autopilots to facilitate self-driving UAVs. The second step involves upgrading the pipeline so that it can run directly on edge GPUs on UAVs to realize fully automatic autonomous online inspections.

Want to know more about how we work with deep learning and UAVs? Read the full article here.