The application of machine learning in a pilot project with Google Cloud and SoftServe moves classification one step closer to condition-based maintenance, writes ABS director of technology Gu Hai
Image recognition powered by artificial intelligence (AI) is a proven technology in many fields, such as medicine, transportation and security.
In the maritime industry its potential is just beginning to be recognised and applied.
In non-marine fields, the objects that the technology is asked to analyse tend to have well-defined characteristics in terms of shape or colour. In shipping and offshore energy applications, the anomalies that need to be detected and analysed are more random in shape, pattern and colour, which will make their assessment more complex.
Potential visual impairments – such as mud or fouling on floating LNG (FLNG) vessels, floating production storage and offloading vessels or offshore platforms, different types of hull and structure coatings and difficulty of access and uneven lighting – add to the challenge.
About four years ago, class society ABS began work on the use of image recognition to support visual inspections with a Singapore university, in a joint development project (JDP) to develop a proof of concept in a marine environment.
Building on this JDP, ABS began to look for potential industry partners to further develop this capability and ultimately make it available for commercial applications.
Our concept was that a machine learning-based image recognition tool should be able to identify and detect anomalies in the ship or offshore structure and quality of coatings from photographic data alone. This would assist surveyors in assessing the development of corrosion or coating failure and also lay the groundwork for a more predictive approach to maintenance.
ABS shared its ideas about the use of AI-based machine learning with a number of companies, including Google Cloud, which saw this as a good application for its Cloud AutoML and open-source machine learning package, TensorFlow, in a test environment.
With an initial agreement in place, ABS could undertake a feasibility study on the basis that if Google Cloud chose to proceed, it would provide a product via its third-party platform partner, SoftServe.
The pilot project ran for about one month, using photos taken from both ships and offshore platforms by surveyors on regular class inspections.
Typically, a surveyor must manually assess a structure for corrosion or coating breakdown and give a rating of the condition. Whether the coating is in good, fair, or poor condition is the driver to whether the owner needs to act.
Current practice combines the surveyor’s experience with simple charts or other reference material to generate a rating on the coating’s condition. The rating is based on the percentage of the area that is corroded, but the surveyor does not physically measure this area of corrosion, instead using their judgement to make a determination.
The image-recognition tool ABS designed can be calibrated to detect the total area of corrosion or coating breakdown and calculate the area pixel-by-pixel, to give an accurate and consistent report on the corroded area. This means that it can be used to analyse any type of marine structure onshore or offshore, provided it is trained with sufficient data to make the assessment.
One of the drivers for the development of this technology is that in future more and more inspections will be made remotely, using technology such as drones or crawlers.
Remote inspection technologies include remotely operated platforms or vehicles such as drones and crawlers, platforms which are used to inspect hard-to-reach places, such as confined spaces, at height, and underwater.
In addition to reducing the risks associated with sending surveyors inside tanks or other confined spaces, these technologies generate a far greater volume of photographic data to analyse and interpret.
Because the ABS in-house image library has a sufficient volume of information, the pilot project used existing photographic data sourced from ABS surveyors.
Teaching the neural network
To make this type of AI work accurately the machine learning tool must be trained with sufficient data to recognise different types of coating failures and structural components, such as hull stiffeners and plates.
This machine learning tool is in fact a type of ‘neural network’. The normal process of employing this type of tool is firstly to identify the most appropriate neural network because there are many different versions available, each designed for a different purpose.
After testing a number of neural networks, we selected the most suitable for the pilot study and trained it with data; in our case photos of corrosion. A neural network has many parameters, making it is necessary to adjust the parameters in training in order to get it to respond with the information we wanted to know.
To accurately identify the presence and location of corrosion or coating breakdown on the image, all the photos needed to be tagged, requiring intervention by an ABS engineer to examine each image and mark the affected area, noting the extent and severity of the corrosion. This would tell the neural network where the corrosion is visible and how serious it is.
This meant tagging a large number of photos to cover all the scenarios we might find on a ship or other structure, not least because there are many different types of coatings in multiple colours and the degree of corrosion can be different based on the level of exposure.
For each neural network in the pilot, we used a few hundred photos, 80% of them for training and the other 20% for testing the accuracy of its answers.
Once the neural network was trained, it was able to detect and recognise corrosion by taking a raw photographic input, detect the corrosion on the image and overlay the corroded area in a different colour on the image.
Because the neural network was able to identify the location of the corrosion it could calculate the corroded area pixel-by-pixel, providing a rating using criteria from ABS Rules based on its evaluation percentage.
The pilot project with Google Cloud and SoftServe delivered promising results and the next stage will see further in-house testing and evaluation by ABS. A timescale for full scale commercialisation or when the system will be employed in the field has not yet been finalised.
What is certain is that this type of machine learning moves ABS towards its stated goal of offering condition-based surveys and being able to provide more predictive analysis as part of the class inspection process.
This ‘condition-based class’ approach is made possible because of AI’s ability to consistently analyse images which enables the prediction of the future condition of coatings and corrosion by analysing historical data.
If enough data can be accumulated using multiple images of the same area of coating, then over a period of time it will be possible to make predictions based on its likely future evolution.
In being able to bring a predictive approach to asset management, ABS believes that AI-driven inspections on ships and offshore assets bring potential benefits for safety, accuracy and efficiency.