Gesture recognition for vessel control

Future ship bridges will have gesture recognition, enabling officers to command ships without touching any instruments. Watchkeeper gestures could be quicker and more effective than using a joystick, propulsion control, mouse or even a touchscreen.

Contactless solutions would be useful if a touchscreen is not available, controls cannot be used or if bridge displays are kept in low-light mode for night operations.

Speech recognition and interpretation allows contactless control in quiet working environments, but ship bridges are often too noisy for this technology.

An alternative approach to provide non-contact control and command is being developed and tested in Germany. This requires a three-dimensional recording of distance, movement and gesture for man-machine communications without using light.

Fraunhofer Institute for Photonic Microsystems (IPMS) has developed a new class of ultrasonic transducer to reliably detect distance changes, movement patterns, and gestures in ranges of up to 50 cm. It has applications in vehicle control, but could be adapted for vessels.

Simple hand movements such as wiping, pulling or pointing could be recognised for commands, similar to the way these are recognised for operating mobile computers and devices.

Ultrasound science

Fraunhofer IPMS researchers have developed a micro-chip architecture that can generate and receive ultrasound in frequencies up to 300 kHz. Reflected sound waves are analysed by measuring the time it took the wave to travel between the sensor system and the reflecting object. They are assessed for the shift in frequencies due to the Doppler effect.

Evaluating the ultrasound provides a spatial resolution for natural movements and gestures in a millimetre range at distances up to 50 cm.

This technology has advantages over using other methods of identifying movements for control, says Fraunhofer IPMS group leader Sandro Koch.

“Compared to camera-based systems, our ultrasonic sensors enable the construction of significantly cheaper electronic and software systems due to longer signal transit times,” he says.

“Our transducers are not susceptible to stray light and allow for reliable data acquisition on optically transparent surfaces. Our systems are more compact and can be inexpensively produced in mass quantities.”

Fraunhofer IPMS are incorporating a new class of electrostatic micro-electro-mechanical (MEMS) bending actuators in the gesture recognition. This was further advanced for generating sound in different frequencies in micro-loudspeakers.

Fraunhofer IPMS’ proprietary nano-e-drive (NED) principle use the high forces of electrostatic fields in nanometre-sized electrode gaps, which allows for mechanical movements with displacements in micron ranges. The chip surface and complete component volume is used to generate sound.

“Using the entire chip volume for sound generation enables us to produce very small components,” says Mr Koch. “Because hundreds of such devices can fit on a single wafer and multiple wafers can be simultaneously processed in single process steps.” This minimises the cost of manufacturing large volumes of these chips.

Future developments

The next step for Fraunhofer IPMS is to ensure high sound pressure, which is converted from air flow, can be adjusted to an increased signal-to-noise ratio for low-frequency ultrasonic transducers.

The resonance frequency, detection range and spatial resolution can then be defined by the geometry of the NED bending actuators.

This gesture recognition technology could be combined with developments in bridge window displays, incorporating augmented reality graphics for a workstation-free and even hardware-free vessel bridge.