Anomalies detection in asphalt pavements : a morphological image processing approach

The inland transport system hold great importance in modern life, but maintaining each kilometer of road network is a difficult task given its extension. The assessment of road conditions may be a tiresome task without automated computational aid; therefore, semi-automated road analysis systems were created. In this paper, a new vision based algorithm for road anomalies detection, such as potholes and cracks, using morphological image processing based on the statistical confidence intervals concept is proposed. The algorithm was tested using confidence intervals ranging from 0.1% to 10.0% in a set of 186 pictures, that where pre-processed by a road segmentation algorithm. The best suitable confidence interval value was 4% for controlled scenarios – where there is no automobiles running on the road and there is no significant shadowing – resulting in 84.00% of accuracy, 75.68% of precision and 90.32% of recall.


Introduction
The inland transport system hold great importance in modern life, it influences since citizens private lives to the economy system.The automobiles, including motorcycles, cars, trucks, buses etc., are one of the most important modes of transportation.However, to this mode of transportation work properly, roads are required, whereas the roads conditions are directly related to the trips in both quality (comfort and safety) and performance (travel more in less time).Supposedly the roads were first designed and built by the Greeks in order to facilitate army's movement and create better trading routes.Originally, the roads were basically an open path (without vegetation) on dirt road.Nowadays there are the paved roads, one of the types most found in cities and highways, connecting neighborhoods, cities and even countries.These roads often consist of asphalt, concrete, or composite pavements [1].Although this type of road improves the quality and performance of trips made by automobiles, such roads may also present some problems, where potholes and cracks seem to be the most common.
In order to monitor the roads condition, several road network maintenance programs have been established, assessing roads condition [2].To cite two cases, there is the Long-Term Pavement Performance program in United States, concerned with data collection, storage and analysis of the United States and Canada road network [1], and a Google® initiative in Poland on mapping potholes, using the mobile devices accelerometer to detect shocks induced by potholes while traveling by automobile.The collected data is sent to Google, which displays this additional information on Google Maps.
The three most frequently approaches on detecting road anomalies are: 3D reconstruction-based, vibration-based and vision-based methods [1].In the first approach, the laser scanning (used to generate 3D point clouds) and stereo-vision, seems to be the most used techniques.However, this kind of approach is costly and requires high computation efforts, hampering its use in real-time approaches.The second uses primarily accelerometers and microphone to detect the vibrations based on the vehicle mechanical responses [3].This technique requires small data storage and it can be real-time processed.However, one needs to pass through the road anomalies in order to detect it, i.e. one have to damage his automobile on a pothole to generate the required data to the detection.In the third approach, 2D vision images are used to segment the road anomalies from the regular roads.This method however, is affected by the illumination level, shadows, climatic weather, car obstruction and overall image quality.
The collected road images used in vision-based approaches are commonly assessed by technicians without computational aid, a time-consuming and costly task that may be influenced by subjectivity and personal experience.There are already some researches using automated systems for road problems detection, which have the main objective of reduce the number of pictures, leaving only the critical images for technical evaluation [1].With the same objective, this paper presents a new approach using morphological image processing for road anomalies (cracks and potholes) detection.In order to present the developed algorithm, this paper was organized as follows: in Section 2 some related work are presented; in Section 3 the road segmentation algorithm and the road anomalies detection algorithms are presented; in Section 4 some test results are shown; and Section 5 contains the conclusions.

Related work
As mentioned before, there are basically three different approaches used in road distresses detection: 3D reconstruction, vibration and vision techniques.There are several studies using each of these approaches.Each resulting image is classified as: true positive (TP), when there is a characteristic and it is correctly predicted; false negative (FN), when there is a characteristic but it is not predicted; false positive (FP), when a characteristic is wrongly predicted; and true negative (TN), when a specific characteristic not occurs and it is also not predicted.With these four values, an approach could be assessed by three prime information (1): accuracy, the number of correct predictions divided by total number of predictions; precision, the number of correct positive predictions divided by the number of positive predictions; and recall, the number of correct positive predictions divided by the number of real occurrences. (1) In a study, [4] introduced a laser-based optical system for pothole detection on roads.This method uses a camera for capture images from laser beams onto the pavement surface, and an image processing algorithm for identify the distresses.Through a laser beam line targeted on the road, one can determine if it is a good condition road (a straight line approximately) or a pothole (a curved line).As this process involves image data, it is influenced by external lighting.[4] also developed a distress classification using a three-layer feed-forward neural network, based on histogram analysis.They perform an experiment with a set of over 100 images, ten images for each distress, but their result was superficially presented.
Using a different approach, [3] developed and tested a detection system for mobiles, using its accelerometers, called G-ZERO (use all three axis values from accelerometer data to detect a pattern).In their study, they also had reviewed and tested other three algorithms -Z-THRESH (use of threshold levels for segmentation), Z-DIFF (use the difference between two consecutive measurement values for segmentation) and STDEV(Z) (use the standard deviation value to segment regions with great variation) -which as well use accelerometer data, in a test track of 4.4 kilometers long, with four different mobile devices.The algorithms were run with a list of different threshold parameters and the best result was chosen for a comparison.The Z-DIFF algorithm has shown the best result with a 92% rate of true positive detections.
On a visual-based approach, [1] presented an algorithm for automating the process of pothole detection using visual pavement data.They consider three basis principles: (A) potholes have shadows that are darker than the environment (road), (B) potholes normally have an elliptical shape in a perspective view and (C) the texture inside a pothole is coarser and grainier than the normal pavement.
In [1]'s algorithm, the collected image is transformed to gray-scale, then a 5 by 5 median filter is used to reduce some noise, and a triangle algorithm (from [5]) is applied over the image histogram in order to segment the road anomalies from normal road (according to the principle A), creating a binary image.This image is then assessed considering the shapes and sizes of the segmented regions.Using the major axis length, centroid position and orientation angle, each connected component is assessed as a partial pothole or entire pothole candidate.When partial potholes are found, the morphological thinning is used to shrink the shade region to a skeleton, then an elliptic regression results in the probable entire pothole region (according to the principle B).At least, the texture inside a pothole candidate is assessed through standard deviation from inside pothole candidate texture and normal road texture.
For testing the algorithm, [1] used a collection of 70 road pictures taken with robot aid, simulating an ordinary passenger vehicle equipped with a high-speed camera.The created picture database had 30 distress photos which contained one or two potholes each.For this set of images, the algorithm accuracy was 85.8%, its precision was 81.6% and its recall 86.1%.
Intending to detect road cracks, [6] proposed an algorithm using morphology.Considering properties of cracks and making use of dilation, erosion, opening and closing, the algorithm was implemented and tested with 66 gray scale sample images, which 44 had cracks, resulting in an accuracy of 95.5%.
Although there are several visual-based approaches most of them use pictures taken from good perspectives that, most of times, have its focus on the anomalies; this, however, might be difficult to do in real case scenarios.Perhaps the plainest method to photograph roads for analysis is an approach like Google® did in order to collect data for its application Google® Street View: use of a camera embedded in a car that whilst travels by the roads, also takes pictures of it.The algorithm presented in this paper was conceived aiming the use of these kinds of pictures in a new visual-based approach.

Algorithms
In order to automate some activities in the road distress detection, a new morphological visual-based algorithm, to detect paved asphalt road anomalies such as potholes and cracks, is proposed.In this new approach, one established requisite was to simulate a real scenario, where pictures are collected by a camera embedded in a car, instead of using pictures taken from a specific road segment.For this reason, two algorithms are presented: one for the road segmentation (the travelway), and another for road anomalies detection.

Road segmentation algorithm
The first idea to segmenting roads was to use the street tracks, however this approach would not be robust enough, since there are some roads without it or eroded center lines and sidelines.The adopted approach was to perform a segmentation using a conditioned Watershed algorithm [7] with pre-defined markers, and considering the road position at the center.
First of all, the RGB picture (Fig. 1-A) is converted to a gray-scale image -following the Advanced Television Systems Committee Standard (Y = 0.2126R + 0.7152G + 0.0722B) -and cropped (Fig. 1-B), 50% in height, removing the highest parts of the picture (e.g.sky and background natural environment), and 40% in width, removing 20% in each side (e.g.buildings, another roads, sidewalk, natural environment).Then, to remove some contrast peaks from the image (Fig. 1-C) the morphological HMAX algorithm [7] was used (with H=25).As the road should be at the image center, bigger in the bottom and smaller in the top (i.e.perspective view), a frustum marker is created with 30% image height and 40% image width as a seed for the road; and two inverted rectangular triangles are created in the upper borders with 33% image height and 15% image width as a seed for the environment (Fig. 1-D).Adopting these markers, the conditioned Watershed algorithm is used considering 4-neighborhood connectivity, in order to get the watershed lines that supposedly separate road from environment (Fig. 1-E).Then, using a morphological reconstructive algorithm [7] with the watershed lines and the same central maker as input, the result will be the binary segmented road.To obtain the RGB road again with a white background (in 8-bits) the equation ( 2) could be used (Fig. 1-F), where rec stands for the binary segmented road reconstructed, pic stands for the preprocessed picture, and neg stands for the morphological image negation.
Note that in some cases the solid center line is so strong that the watershed segmentates just one side of the road whereas the other side is embedded in the environment segmented region.Nevertheless this is not so bad because the most important track for anomalies detection is the drivers track and anyway, to detect anomalies in both sides, the car just should drive the same path in reverse.Furthermore, the determined frustum dimensions was suggested considering the roads in which the opossite car flux is at the drivers left or at the drivers right (most common in Europe), otherwise a bigger frustum could be used to get the whole road.
Two problems were noticed when using this approach: (a) when the road anomalies are extended until the road borders, these anomalies could be segmented as environment and not as a road space; (b) when the sidewalk has a very similar color to the road and there is none perceptible divisor (between the road and the sidewalk), the first could be segmented as a road and not as an environment component.

Road anomalies detection algorithm
Owning a segmented road image with a white background (Fig. 2-A -road_image), the first operation is to crop again some borders off the picture.In this approach, a frame with 66% of the image center is used, while 34% is cropped, reducing the image area but keeping most of the road (Fig. 2-B).At this point one [1] basis principle (A) was used, considering that potholes have shadows that are darker than the road.Actually, not just the potholes have darker areas but also the cracks.In this way, a histogram analysis seems to be appropriated in order to distinguish road from potholes and cracks.But before generating the histogram, basins contrast are removed (Fig. 2-C) with a morphological HMIN algorithm (with H=10).
As the road should be the only element on the image, it was assumed that the greater histogram value should belong to a road pixel intensity, hence after called road_index.Analyzing the histogram curve, the supposedly anomalies would have the darker pixels intensities.Considering it, the triangular histogram algorithm approach from [5] (like in [1]), was tested; however, the obtained results were not so good: often the gray scale intensity was too high, segmenting also parts of the road.
Assuming the histogram curve similar as the Gaussian distribution (Fig. 3), and that, regular roads would have lower standard deviation than distressed roads, a new method was proposed using standard deviation and (2) confidence intervals (C.I.).Ignoring the white background in the computation, one value (called n_dev) was chosen to determine how many standard deviations away from the biggest pixel value should limit the segmentation.Using the histogram standard deviation (std.dev) and n_dev values, the threshold index could be calculated (3).
After segmenting the road image with the threshold value (img) (Fig. 2-D), a morphological closing operation is executed with a disk, with radius 0.001% image pixels (D 0.001% ) as structuring element, in order to connect nearby components (Fig. 2-E).Then, a morphological area open considering 4-neighborhood is used to remove small components (possible noise), maintaining only the connected components of 0.05% image pixels (a 0.05% ) or more (Fig. 2-F).Briefly, these operations can be expressed according to (4).
After detecting the anomalies, the next step of the algorithm is to delimitate better these structures, using a new watershed, and to highlight them on the image, using border lines.In order to use a new watershed, an external marker is created using the morphological negation of the dilated segmented anomalies image (anomalies'), using a disk of radius 0.05% image pixels as structure element.Furthermore, an internal marker is also created performing a morphological erosion operation, using an one pixel cross as structure element, on the segmented anomalies image (anomalies').
Holding both external and internal markers (Fig. 2-G), a new conditioned watershed operation is performed over the road_image (using 4-neighborhood) getting the watershed lines (Fig. 2-H), followed by a morphological reconstruction using the internal marker as seed (Fig. 2-I).Finally, a closing operation, using an one pixel cross as structuring element, is performed in order to connect (with 4-neighborhood) components that were connected by an 8-neighborhood (Fig. 2-J).The resulting binary image is then used to generate the borders that delimitate the detected anomalies through the difference between this image and its dilated form, generating a final image (Fig. 2-K).

Tests and results
A prototype version of the proposed algorithm was implemented using the morphological algorithms provided by the SDC Morphology Toolbox for Python called pymorph 2 .Concerning to the sample pictures used in the tests -rather than create a personal road pictures database -images from Google® Street View were used, since they fullfil the stated requisite (photos taken by a camera embedded in a car).In order to collect a set of pictures, a script was created using also the Google® Directions API, where giving two coordinates a route h ad been calculated by Google® Directions API, providing drive instructions throught a XML file, which was used as input for getting pictures from Google® Street View in the better quality available: panoramic pictures with 13,312 x 6,656 pixels.However, to seem a normal picture taken by a regular camera embedded in a car (not panoramic), just a frame of 5,120 x 2,048 pixels -the front view -was caught.A total of 229 images were collected from Google® Street View, and were splited in four groups: crack pictures with 63 samples, pothole pictures with 13 samples (besides having potholes, some of these pictures contained also road cracks), normal road pictures with 123 samples, and shadowy road pictures with 30 samples.
Since the used pictures contains both the roads and the backgrounds, the road segmentation algorithm was firstly used and assessed.The results are presented in Table 1, which shows the segmentation quality for each group of pictures, classifying them in good segmentation -when at least one quarter of the entire road was segmented (even when part of the environment was also segmented) -and poor segmentation -when less than one quarter of the entire road was segmented.Table 1 also shows the number of unusable pictures due to three main reasons: (1) the segmented road had poor quality, (2) environmental objects (such as buildings and sidewalks) were also segmented, and (3) the anomaly was not segmented (only for potholes and cracks pictures).The results show that the segmentation quality was good in 92.14% samples, while 81.22% (186, i.e. pictures minus unusable pictures) of the processed pictures could be further analyzed for anomaly detection.
Qualitatively, some results were excellent, where the entire road or only the driver track was totally segmented.These good results seemed to be regular when the roads were well demarcated by sidelines.
The 186 usable segmented road pictures were further used in an assess trial for the proposed road anomaly detection algorithm.The algorithm was executed with a set of values to the number of standard deviation (n_dev) based on confidence intervals, ranging from 3.100 standard deviations (less restrictive)representing a confidence interval (C.I.) of 10.0% in the Z table -to 1.281 standard deviations (more restrictive) -representing 0.1% in the Z table (C.I.).The trial results, for each group of pictures, are shown in Tab. 2, where crack and pothole values are related to true positives (TPs) -at least some road anomaly fragment was detectedand false positives (FPs) -another object was detected instead of a road anomaly fragment -while normal road and shadowy road values are related just with FPs.As none pothole picture hit FPs, this column was omitted.The results in Table 2 show that increasing the n_dev value, the TP hits for cracks and potholes, and the FP hits in normal roads and shadowy roads increases.Road cracks and normal roads were the two most dependant on the C.I. value, presenting an increase of 75.86% (TPs+FPs) and 35.09% (FPs) detections respectively, when comparing C.I.=0.1% with C.I.=10.0%,whereas the potholes seemed to be the less dependant on the C.I. level since even with C.I.=0,1%, eight from nine potholes were detected.
Concerning to the crack FPs, most of them occurred because of automobile detections.For instance, with C.I.=0.1%, three from five FPs were related with car detection (the other two were related to cars, but also to dirt road).In fact, every car present in the segmented road picture was detected.Furthermore, the TPs were also affected by the anomaly distance, since the images are in perspective view where close regions are bigger and farther regions are smaller (e.g. one pothole was not detected in some cases because of its distance from the car/camera).Some FP hits in the normal road image group have ocurred, even with low C.I. level, because of swales, road gutters, road stains, shadow from road cat eyes and tire marks.Even some thin cracks not sawn with naked eye were caught, but considered FPs because of its small size.Regarding to the shadowy road pictures, as expected, plenty of FPs ocurred, since this approach uses as basis the anomalies lighting traits.However, when the road is mostly covered by shadow, the road_index is smaller, whereas just very dark pixels are selected as anomaly, ignoring shadows with brighter intensity.
Qualitatively, using C.I. > 5% makes the detected anomaly regions spreads into other normal regions, and also starts to include lots of regular regions, decreasing the algorithms precision.With few exceptions, lower C.I. levels produced lower recall but higher precision and accuracy, while higher C.I. levels produced higher recall but lower precision and accuracy.
To show this relations, Table 3 is presented with the recall values, three precision columns and three accuracy columns.The overall precision and overall accuracy columns show the precision and accuracy respectively, regarding to all 186 results obtained; the precision' and accuracy' columns show the precision and accuracy in a controled scenario, where there is no automobile in the picture (i.e.ignoring the segmented images with automobiles); the precision" and accuracy" columns show the precision and accuracy in an even more controled scenario, where there is no automobile and shadows in the picture (i.e.ignoring the segmented images with automobile and the shadowy roads group of pictures).As shown in Table 3, the overall precision was not so good, but in a controlled scenario without automobiles traveling in the road, this value increases substantially -more than 20% in some cases.Considering an even more controlled scenario, without automobiles traveling in the road and without shadows on the road, the precision increases even more, reaching the max value of 95.24%.However, the overall precision seems to be less affected by higher C.I. levels when compared with the cogitated controlled scenarios, whereas the first had a max difference in precision of 8.22%, whilst the second scenario had 21.90% and the third had 34.63%.Despite presenting good results, these controlled scenarios were not easily obtained.
The accuracy also presented an increase when comparing the general scenario with the controlled scenarios.Nevertheless, the accuracy seemed to be less affected by the C.I. levels ranging from 0.1% to 5.0% in all scenarios, with small variation.But the transition from C.I. 5.0% to C.I. 10.0% generates substantial accuracy variation.
The appropriate value of C.I. used in real events may vary according to the requirements: while in one case the recall could be of utmost importance, in other, the highest precision could be sought.But considering the best C.I. value as the higher combination (biggest sum) between accuracy, precision and recall, this experiment indicates that the values 4% (to controlled scenarios) and 4.5% (to general scenarios) are the best values for C.I. level to the proposed algorithm.

Conclusions
Although vibration-based approaches have their advantages, the need of passing through road anomalies to detect them is a significant drawback.Therefore, visual approaches seem most appropriate, whereas the biggest problem of these approaches is external factors such as weather conditions and shadowing.Also, detecting algorithms should have good balance, since on a very restrictive approach any non-predicted standard can be detected as an anomaly (FP), while a slightly restrictive approach can miss some road anomalies (FN).
In a real environment, where pictures are taken by a high-speed camera embedded in a car, the proposed algorithm for road detection using watershed algorithm showed good results; in 92% of the pictures the travelway was total or partially (at least one quarter) detected.For further studies, the use of different parameters, such as watershed markers, could be evaluated.
Regarding to the road anomalies detection algorithm, its results show that the C.I. value should be chosen based on the desired requirements, since there is a trade relation between recall and precision: moderate recall with high precision or high recall with moderate precision.Its results also showed few FP cases when using low

Figure 1 :
Figure 1: Example of road segmentation -(A) original image, (B) cropped image in gray-scale, (C) use of HMAX algorithm, (D) image with watershed markers, (E) watershed lines, and (F) segmented road

Figure 3 :
Figure 3: Similarity between a segmented road image histogram and the Gaussian curve

Table 1 :
Road segmentation results

Table 2 :
Results of road anomalies segmentation

Table 3 :
Road anomalies detection algorithm accuracy, precision and recall, in general and controlled scenarios.