Machine Learning Applications in Remote Sensing 🛰️

Leveraging machine learning in the field of remote sensing has opened up new avenues for interpreting complex patterns in satellite imagery. While accessing and analyzing such data may not always be straightforward, the community has managed to develop models for a wide range of applications, including monitoring land changes and comprehending climate fluctuations. In this section discover the full range of machine learning applications supported by the ML-STAC specification.

Tensors vs Images 🤔

Within the ML-STAC specification, we opt to use the term "tensors" instead of "images" when referring to individual data points. But why this distinction?

Although a significant portion of available EO datasets are indeed image-based, in the real world, EO data is much more complex and intricate. The term "image" typically refers to the conventional three-channel RGB imagery associated with typical computer vision. However, in the field of Earth Observation (EO), images might have several channels, capturing a broad spectrum of information.

Moreover, remote sensing studies frequently don't limit themselves to single timepoint snapshots. They analyze several temporal iterations of the same site, providing insights into shifts and transformations over extended periods. Sometimes, datasets might even involve intricate multivariate ensembles of model simulations.

In summary, the data in EO can span from simple 1D structures to intricate 5D compositions. Given this complexity and the goal of establishing a comprehensive specification adaptable to diverse challenges, the term "image" felt too restrictive. Hence, "tensor" offers a more fitting and inclusive descriptor.

Order (L)	Set of tensors C(L)	Description of C(L)
0	`C({})`	Scalar value where no dimension is defined
1	`C({lat})`	Univariate latitudinal profile
1	`C({long})`	Univariate longitudinal profile
1	`C({time})`	Univariate time series
1	`C({var})`	Single multivariate observation
2	`C({lat, long})`	Univariate static geographical map
2	`C({lat, time})`	Univariate Hovmöller diagram: zonal pattern over time
2	`C({lat, var})`	Multivariate latitudinal profile
2	`C({long, time})`	Univariate Hovmöller diagram: meridional pattern over time
2	`C({long, var})`	Multivariate longitudinal profile
2	`C({time, var})`	Multivariate time series
2	`C({time, freq})`	Univariate time–frequency plane
3	`C({lat, long, time})`	Univariate data cube
3	`C({lat, long, var})`	Multivariate map, e.g. a global map of different soil properties
3	`C({lat, time, var})`	Multivariate latitudinal Hovmöller diagram
3	`C({long, time, var})`	Multivariate longitudinal Hovmöller diagram
3	`C({time, freq, var})`	Multivariate spectrally decomposed time series
4	`C({lat, long, time, var})`	Multivariate spatiotemporal cube
4	`C({lat, long, time, freq})`	Univariate spectrally decomposed data cube
5	`C({lat, long, time, var, ens})`	Multivariate ensemble of model simulations

Sets of tensors available in the EO domain. Adapted from Mahecha et al. 2021

ML Tasks in Earth Observation 💻

This section explores the wide range of Earth Observation (EO) datasets available within our community. These tasks are broadly classified into two main categories: Remote Sensing tasks and Multi-Modal tasks.

Remote Sensing tasks encompass those where the input is derived from an EO data entry point, and the output is a valid tensor object. A valid EO data entry point must be characterized by its spatial and temporal dimensions, either as dimensions or metadata.

Conversely, Multi-Modal tasks involve integrating EO tensor data with other types of data, such as textual, audio, or other sensory data, which is not easily integrated directly in a tensor format. The input and output may not necessarily be a tensor.

In the following subsections, we'll delve deeper into each category, highlighting the challenges and the proposed unique solutions to organize datasets in the ML-STAC specification.