Pontus Boström is a machine learning engineer at Groke Tech, who joined the company after working in academia for many years. A growing interest in data science guided him to Groke, where he now focuses on computer vision, but still indirectly harnesses many of the skills he acquired during his time as a researcher. This week, we spoke to him about how Groke’s situational awareness system, Groke Pro, can help mariners navigate the seas safely, both during the day and at night.

One of the key features of Groke Pro can be found in the fact that it can be used to aid navigation at night. The system gathers data from various sensors to detect objects in the vicinity of the ship, then uses software to combine all of the information together and provide an augmented reality (AR) view of a ship’s surroundings.

The AR view is formed from a video stream that captures a part of the surrounding view in visible light. This view is then overlaid with another feed of the same perspective but captured using a thermal camera.

Boström explained how thermal cameras work by harnessing the properties of infrared light.

Thermal cameras are used in night vision applications because they detect wavelengths longer than visible light. Those wavelengths correspond to the amount of heat being emitted by an object, so even in the absence of visible light, a thermal camera can detect objects that are nearby because of the thermal radiation they emit.

“In the Groke Pro application,” Boström said. “The visual and thermal images are combined to give the system the best of both worlds.”

At night, watchkeepers can look out the windows of the bridge, where they will be able to see various sources of light, from passing ships and shore roads, to the stars in the sky. The problem here is that they won’t be able to see much else in the visible spectrum, and point sources of light are notoriously difficult to determine the distance to. That is part of why navigating at night can be dangerous.

But this is where the thermal camera comes in.

Because of residual heat in the water, ships, buoys, buildings on shore, and the land itself, a thermal camera can capture the outline of the horizon, ship silhouettes, and other objects. When that thermal camera feed is blended with the visible feed, watchkeepers are provided with a more coherent view of their surroundings.

Boström showed us a few sample images of unblended and blended camera views, and the difference was striking.

One of the images showed the view of a dock at night. In the visible light image (taken in color), we could see bright white lights from a distant shore facility, the reflection of those lights in the water that also reflected in the water, and some more subdued orange lights from the dock itself. However, nothing about position or distance could be determined.

The infrared image (in gray-scale) then showed where the water met the dock and the horizon in the distance, but more importantly, it also showed that a ship was tied up to the dock. That ship was essentially invisible in the visible light photo, as its lights could not be distinguished from those on shore.

It is only when the two views are blended that the whole picture comes together. The color imagery from the visible feed showed us the lights of the shore facility, dock, and ship, and what color each light was – something that is very important when navigating the seas. Meanwhile, the gray-scale thermal imagery showed the shape of the ship, the distance to the shore, and other positional information.

Boström explained that Groke’s aims for near real time operation as possible, so that what you can see in the AR view and looking out the window would match, and eventually, they are also looking at integrating other sources of data as well to make the system even more reliable.

We ended our discussion with Boström by asking him about his feelings about being a part of Groke and having a hand in building the Pro solution.

“For me, this is a really interesting project because I get to work with really interesting technologies and state-of-the-art algorithms, and try to build a system that works in practice, not just as an academic exercise,” he said.

“Also, there are challenges that come along when you don’t have to just make it good enough for a (research) paper, but how to make it work in the real world.”

That is a sentiment reflected by many who now work with Groke – a desire to create something that will have a real effect on making ocean navigation safer for all.

Video on how blended view works in sea mist:

Article published originally in Groke’s exclusive customer newsletter in November 2021