Objectives of the service
In-terra-gate is a software that combines large language models (LLMs) and visual foundation models (VFMs) to allow the direct interaction with satellite data using natural language. This allows users with less experience in handling satellite data to take advantage of the benefits and potential of geospatial data by expressing the desired information in natural language, which is translated into an image processing task. The technical part of the activity focuses on the adaptation and fine-tuning of an existing pre-trained LLM and VFM for satellite imagery and the training of a fusion module in order to create joint textual and visual embeddings. For the demonstration of the functionality a chatbot user interface is developed which allows users to select an area of interest and ask questions about geographic and geological characteristics. The business part of the activity focuses on validating the target customer group by conducting a market analysis and establishing potentially interested contacts as well as determining their needs and requirements. This ensures that the development of the software is geared towards market requirements and incorporates user and customer perspectives at an early stage.
Users and their needs
The main customer target group is the insurance sector where the users are underwriting managers.
There is a letter of support from Santam, a large insurer from South Africa, expressing interest in
natural language based satellite data analysis for building and property risk management. Further
user engagement shows that geodata and geoinformation tools for the risk management of buildings
and properties have so far been little used in the insurance sector. Where underwriting managers
already use geodata, it is clear that a comprehensive geodata basis is required on the one hand and
simple and intuitive interaction with it on the other. In the course of discussions with insurance
companies, the following user requirements were identified:
-
The user requires the vision model to identify the style, material, dimension and orientation of the roof of a building and the distance to nearby buildings, vegetation and water bodies
-
The user requires the system to identify whether the property lies in a natural risk area
-
The user requires the vision model to identify existing infrastructure and buildings in residential areas
-
The user requires the model to translate the user query to an image analysis task and to output the result as text
-
The user needs an intuitive and simple to use tool to select map regions of interest and obtain corresponding satellite images
-
The user requires the frontend to have a functionality to integrate and visualise additional data sources
In addition to the talks with Santam from South Africa, discussions were held with companies from Germany, Switzerland, Spain and Australia. Through the user engagement it could be verified that there is a need for providing geospatial information to a wide range of non-technical users by offering intuitive and interactive natural language based access to complex satellite data analysis. Moreover, In-terra-gate provides aggregated information that can be specifically queried by domain experts for more individual
decision and policy making.
Service/ system concept
The system architecture relies on a modern multimodal approach that integrates a Large Language Model (LLM) and a Vision Transformer based vision encoder to enable seamless natural language querying and interpretation of satellite imagery. The vision encoder processes satellite imagery, extracting high-dimensional feature embeddings that capture spatial and semantic information.
These embeddings are then formatted as inputs compatible with the LLM, allowing both the visual data and the user's natural language query to be jointly processed. The LLM interprets the user query, incorporates the visual context from the encoded imagery, and generates natural language responses. This streamlined integration ensures that the system effectively bridges visual and textual modalities.
This architecture exemplifies the latest trends in machine learning, leveraging transformer-based architectures for both text and image processing, and employing fine-tuning and transfer learning on pre-trained models to adapt to domain-specific satellite data. The system is connected to an intuitive frontend interface, which includes a chatbot for query input and an image display for selecting image input.
Space Added Value
For In-terra-gate the satellite data from the Copernicus Sentinel-2 mission are used because of its open availability and its comparatively high spatial and temporal resolution. The developed methods can also be transferred to other satellite data sources. Computer vision and machine learning algorithms to automatically analyse these data has become standard nowadays. However, currently only technical experts can harness the full potential of remote sensing data and users with limited technical expertise rely on already post-processed geospatial information integrated into sophisticated software systems. In-terra-gate is a service that makes satellite data analysis easily accessible to various users and more interactive for more individualized information retrieval.
Current Status
From a technical perspective, a working backend system for text-image-encoding based on an LLM, VFM and fusion module is developed. Additionally, a demonstrative chatbot user interface is available to showcase the basic functionalities of a natural language based satellite analysis service. These prototypical components will be further developed based on the specific market requirements and data basis of the geodata provider, which will be specified in the further partner engagement.
From a commercial perspective, the team generated six very promising business prospects with whom further exchanges will take place in order to bring the service into productive use. The team concluded that the best market perspective comes from collaborating with a geodata provider and offering standardized remote sensing services with a natural language based interface that can be customized for client specific needs.
