Objectives of the service
Image credit: HoloCa/OICL
The maritime industry is an industry with strict regulations, high demand on
safety and extreme valuable assets at risk. Up to 85% of maritime accidents are claimed to be caused by "human error" and several studies conclude that the design of modern bridges and miss communication is part of the cause. HoloCap has developed a world leading technology for digitalisation and reconstruction of real people as three dimensional “holograms” or volumetric video in real-time. The Kick-Start activity explores how the company's technology and developments in “future internet” such as low earth orbit satellite connectivity and VR/AR can be used in relation to maritime pilotage service. The idea and the objective of the study is to find out if and how HoloCap can develop a remote pilotage system by replacing the physical pilot on the bridge of a ship, with a digital reconstructed hologram. The pilot, on shore, sees the master as if present, receives sensor information for “situation awareness” and can provide pilotage remotely.
If the proposed remote pilotage solution is perceived as a viable, even preferred alternative to traditional pilotage, the solution can disrupt how pilotage is executed forever.
Users and their needs
Two main customer and user groups have been identified in relation to maritime pilotage; Port authority, expert pilots employed by the port authority, ship owner/operator of ship subjected to compulsory pilotage and ship masters employed by the ship owners/operators. The pilot and master represents the users of the proposed HoloCap system and port authority and ship owners/operators customers.
Through dialogue with the customers/users involved in Norwegian pilotage, these main constraints were identified:
- Pilot transportation, even though considered a safety risk, is currently a necessity in pilotage missions.
- Highly expensive operating cost for regulated, compulsory pilotage service. The transportation and logistics involved in bringing the pilot on and off the ship is one of the main cost drivers in pilotage missions.
- There are big seasonal variation in piloting demand
- Communication and cooperation between the pilot and the ship's master is extremely important. Poor communication, often related to non-verbal communication, can result in fatal accidents.
Left: Pilot about to disembark a ship. Source: [Link]. Right: Researchers from HoloCap collaborator Ocean Industries Concept Lab testing the use of augmented reality in arctic maritime operations.
Service/ system concept
The overall goal of the system is to connect the captain of a ship with a remote expert pilot on shore, in such a way that both participants can collaborate in much the same way as they could if they were physically present on the ship's bridge. This is achieved by the use of augmented reality headsets for the users, pluss depth-cameras that can capture a volumetric representation of the participants and optionally, relevant parts of the bridge environment at real-time speeds. To help both participants obtain a shared situational awareness, ship telemetry data and live camera feeds (either from CCTV on the ship or from AR headset), can be streamed as well.
The overall system will then fulfill the regulatory requirement that two “sensors” (which can be either a person, or computer system) are available when performing a remote pilotage operation.
The proposed system uses upcoming LEO satellite connectivity services, to deliver the necessary coverage at sea, within given latency and bandwidth requirements.
The context diagram shows the real-time volumetric telepresence system connecting the two main users, the shipmaster and the pilot. Both have access to ship telemetry data in AR, through a real-time telemetry system. Other external systems which are also used today, can be used by each participant, like radar, AIS (automatic identification system) and ECDIS (Electronic Chart Display and Information System).
Space Added Value
The system will utilize satellite connectivity and low earth orbit satellites capabilities for delivering the remote pilotage service. The application benefits gained by the use of LEO satellite connectivity are safe, consistent coverage regardless of location (remote port, off-shore, open sea or arctic), sufficient bandwidth (dependant upon selected compression), low latency and low packet-loss to facilitate real-time telepresence communication. Other space assets like GPS, wheaterdata and AIS are necessary supportive space assets that are needed for remote pilotage.
Current Status
The Kick-Start activity uncovered the feasibility and market potential of introducing the proposed HoloCap application into the Norwegian maritime pilotage market:
- A need for the service expressed by the users
- Changes in regulations provides no regulatory barriers
- Trends in the industry favours disruptive solutions
- If the value proposition is delivered, substantial cost reduction can be achieved by the current providers of pilotage service. This provides evidence that the market size is large enough to support development/introduction of the service and customer willingness to pay
- A technical solution is viable using a combination of HoloCap`s core technology and emerging infrastructureHoloCap has established relationships with “champion customer” and 3rd party suppliers
- A viable roll out plan for product introduction
- Solid international growth potential
The project success relies on specific developments in the XR space, developments in LEO Satellite connectivity or video compression techniques. These developments in compliance with service criteria needs to be further explored before the project can enter into a demo phase.
A POC was developed during the project and tested in VR
Prime Contractor(s)
