Communication in the maritime environment is important for safety but does not always run smoothly. This can have catastrophic consequences. After all, 90% of all accidents at sea are caused by human error, and half of these by problems with communication. In collaboration with Jade University of Applied Sciences (Elsfleth Campus) and partners from industry, Fraunhofer IDMT in Oldenburg, Germany, has been working for some time on improving safety-relevant voice communication, particularly at sea.

"In addition to developing chatbots for training purposes and recruitment procedures, the focus is on systems for monitoring voice commands on the ship's bridge. For this purpose, we have trained our speech recognizer's AI with special vocabulary, known as nautical terms and phrases, or maritime English," says Jan Wellmann, head of automatic speech recognition at the Oldenburg Branch for Hearing, Speech and Audio Technology HSA of Fraunhofer IDMT.

ELNAV, a start-up company based in the Croatian port city of Split, is developing what it calls a Helm Order Monitor. This electronic device uses speech recognition technology from Fraunhofer IDMT in Oldenburg in conjunction with data received from the ship's sensors and monitors whether the commands issued are clear, confirmed and correctly executed.

Hrvoje Mihovilović, ELNAV's founder and CEO says, "In the beginning, the biggest obstacle for ELNAV was deciding whether to develop our own speech recognition system or find a partner. We needed a robust speech detector that would work reliably even under difficult acoustic conditions. We soon discovered that Fraunhofer IDMT was developing a speech recognition model for maritime communications and therefore an ideal partner for the development of our Helm Order Monitor."

Technological challenges

The partners faced three main technological challenges. The first was the signal degradation caused when using far-field microphones in speech-processing applications. The noise level on the bridge should not interfere with verbal communication, mask audible alarms or be uncomfortable for bridge personnel: the ambient noise level on the bridge in calm weather should not exceed 65 dB(A). To solve this problem, microphone array signal processing is presented as an alternative.

The accuracy of automatic speech recognition (ASR) under different noise conditions and at various distances was a further challenge. To make the ASR system more robust in noisy environments, deep neural networks (DNN) are used to enhance speech.

A third challenge was catering for the endless variations of the English language—from different regional accents to idiosyncratic use of grammar and vocabulary. Here, machine learning is used to create a single, comprehensive language pack which is accurate and encompasses as many variations of English as possible.

"The latest report on European Maritime Safety (EMSAFE) sees digitalization and increasing automation as a major opportunity. However, the technologies could also bring new challenges for safety as well as crew training. The changes required for this are supported by our speech recognition system. It can be used to learn, test and monitor maritime English in practice," says Dr. Jens Appell, head of the Oldenburg branch of Fraunhofer IDMT.