Infinite-Dimensional Vectors

Vector spaces are incredibly useful and, consequently, ubiquitous in modern mathematics. That’s not a coincidence; the modern definition of vector spaces is about 100 years old and the result of about 300 years of evolution. Vector spaces are in fact so useful that physics, engineering, computer science and many other disciplines make use of them on a daily basis to model a large range of problems; from modeling the world in theoretical physics to representing meaning and concepts in machine learning.

If you work in software, engineering, physics, or machine learning, you’ve almost certainly been trained to think of vectors as lists of numbers or arrows. In modern mathematics, that’s not what a vector is, and that intuition collapses the moment you leave finite dimensions.

I hope this article will change how you think about vectors and dimensions by breaking down assumptions about vector spaces. Unfortunately, finite-dimensional vector spaces are not a good basis to build an intuition for infinite-dimensional vector spaces. So let’s forget about $ℝ^{n}$ and start from scratch. I am going to try to beeline into infinite dimensions and ignore a number of important topics about vector spaces.

Vector Spaces

Before we can talk about dimensions at all, we must first stop seeing vectors as lists of numbers or geometric arrows. To do that, we need to start with the mathematical definition.

Definition: Let $V$ be a non-empty set and $𝔽$ be a field (e.g. $ℚ$ , $ℝ$ , or $ℂ$ ).

$V$ is a vector space over the field $𝔽$ if there are two functions

$\oplus : V \times V \to V$ (vector addition)
$⊙ : 𝔽 \times V \to V$ (scalar multiplication)

fulfilling the following axioms for all $u, v, w \in V$ and $α, β \in 𝔽$ :

$u \oplus (v \oplus w) = (u \oplus v) \oplus w$ (associativity)
$v \oplus u = u \oplus v$ (commutativity)
There exists a $0_{V} \in V$ so that for all $v \in V$ : $v \oplus 0_{V} = 0_{V} \oplus v = v$ (neutral element)
For every $v \in V$ , there exists an inverse $- v \in V$ with $v \oplus (- v) = (- v) \oplus v = 0_{V}$ (inverse element)

and

$α ⊙ (u \oplus v) = (α ⊙ u) \oplus (α ⊙ v)$ (distributivity of scalar multiplication over vector addition)
$(α + β) ⊙ v = (α ⊙ v) \oplus (β ⊙ v)$ (distributivity of scalar multiplication over field addition)
$(α \cdot β) ⊙ v = α ⊙ (β ⊙ v)$ (compatibility of scalar multiplication with field multiplication)
$1_{𝔽} ⊙ v = v$ (scalar identity).

It’s common to use regular addition ( $a + b$ ) and multiplication ( $a b$ ) notation to mean vector addition or scalar multiplication based on context. That’s perfectly fine when these operations are not the focus. However, when introducing vector spaces it pays off to be explicit. So for this section, we’re going to use different symbols to introduce the concept and we’ll switch to the regular syntax in the following sections.

What vector spaces are not

It’s important to note that there are a number of things that are reasonable to expect from vector spaces that are not part of the definition of vector spaces.

We don’t have the concept of a distance between two vectors (i.e. we don’t have a metric).
We don’t have the concept of the magnitude for a vector (i.e. we don’t have a norm).
We don’t have the concept of a direction of a vector (i.e. we don’t have an inner product).
We don’t have any general concept of convergence within a vector space.

This means that the very common intuition for vectors of having magnitude and direction doesn’t actually apply to general vectors. There’s a class of vector spaces called Hilbert spaces for which this intuition works, but they are out of scope for this article.

Let’s try to understand what that means by looking at a practical example. My favorite kinds of examples have surprises that challenge preexisting intuitions, so I recommend paying special attention to them.

Example: Let $V$ be the set of positive real numbers $V = {x \in ℝ | x > 0}$ . If we define vector addition and scalar multiplication with $ℝ$ as

$\oplus : V \times V \to V$ defined as $(u, v) \mapsto u \cdot v$ with the neutral element $0_{V} = 1_{ℝ}$ and the inverse element $- v = \frac{1}{v}$
$⊙ : ℝ \times V \to V$ defined as $(α, v) \mapsto v^{α}$

Then $V$ is a vector space over $ℝ$ :

$u \oplus (v \oplus w) = u \cdot (v \cdot w) = (u \cdot v) \cdot w = (u \oplus v) \oplus w$
$v \oplus u = v \cdot u = u \cdot v = u \oplus v$
$v \oplus 0_{V} = v \cdot 1_{ℝ} = 1_{ℝ} \cdot v = 0_{V} \oplus v = v$
$v \oplus (- v) = v \cdot \frac{1}{v} = \frac{1}{v} \cdot v = (- v) \oplus v = 1_{ℝ} = 0_{V}$

and

$α ⊙ (u \oplus v) = {(u \cdot v)}^{α} = u^{α} \cdot v^{α} = (α ⊙ u) \oplus (α ⊙ v)$
$(α + β) ⊙ v = v^{α + β} = v^{α} \cdot v^{β} = (α ⊙ v) \oplus (β ⊙ v)$
$(α \cdot β) ⊙ v = v^{α \cdot β} = {(v^{β})}^{α} = α ⊙ (β ⊙ v)$
$1_{ℝ} ⊙ v = v^{1_{ℝ}} = v$ .

Notice that vector addition and scalar multiplication are not constructed from field addition and multiplication respectively. This is important; vector spaces don’t have to follow the same construction as the standard vector spaces $ℝ^{n}$ .

One thing that’s interesting about this vector space is that every vector $v$ can be represented as $v = λ ⊙ e = e^{λ}$ with $λ = ln v$ . We’ll see later that this means that this vector space has the dimension $1$ .

Basis and Dimension

No more weird symbols

We’re now going to switch to a less noisy syntax. Instead of $(α ⊙ v) \oplus (β ⊙ u)$ we’ll just write $α v + β u$ . However, don’t let that fool you, vector addition and scalar multiplication may behave similarly to their standard counterparts, but they are very different.

Bases are the most important concept to understand vector space dimensions. They are a central concept in vector space theory in general, so it’s well worth having a basic understanding of them.

In short, a basis is a minimal set of vectors that spans the whole vector space. If we have any set of vectors $S \subseteq V$ of a vector space $V$ over $𝔽$ , the span of that set is the set of all finite linear combinations of elements of $S$ :

span (S) = {α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} | α_{1}, \dots, α_{n} \in 𝔽, v_{1}, \dots, v_{n} \in S, n \geq 0}

where $n = 0$ gives the empty sum $0_{V}$ , so $span (\emptyset) = {0_{V}}$ .

For $S$ to be a basis of $V$ , $S$ needs to be minimal and $span (S) = V$ . If $S$ is not minimal, we can remove an element, say $v_{r} \in S$ , such that $S ∖ {v_{r}}$ ( $S$ without $v_{r}$ ) still spans $V$ . This means there’s a linear combination without $v_{r}$ that can represent $v_{r}$ , after all, $v_{r}$ is an element of $V = span (S ∖ {v_{r}})$ :

v_{r} = α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} with v_{1}, \dots, v_{n} \in S ∖ {v_{r}}

This works the other way around as well: If $S$ contains a vector $v_{r}$ that can be represented with a linear combination not involving $v_{r}$ , we can remove this vector and still span the same vector space. This gives us a better way to express minimality: We call $S$ linearly independent if removing any element from $S$ strictly shrinks its span: $span (S ∖ {v_{r}}) ⊊ span (S)$ or, equivalently, no vector in $S$ can be expressed as a linear combination of other vectors in $S$ .

Definition: A basis $B$ of a vector space $V$ over a field $𝔽$ is a linearly independent subset of $V$ that spans $V$ . The elements of $B$ are called basis vectors.

A consequence of linear independence is that every vector in $V$ has a unique representation as a linear combination of basis elements.

You might be wondering why we’re only interested in finite linear combinations and exclude infinite linear combinations. The reason is that infinite linear combinations require a concept of convergence which we don’t have for general vector spaces and even if we did have it, not every infinite linear combination would converge. When we add convergence to a vector space (like in Hilbert or Banach spaces), we can talk about Schauder bases with infinite series. But in general vector spaces, we are strictly talking about Hamel bases (i.e. finite linear combinations).

The reason why bases are so important is that every vector space has a basis and that all bases of the same vector space have the same cardinality.

This lets us define the dimension of a vector space:

Definition: Let $B$ be a basis for the vector space $V$ , then the dimension of $V$ is $dim (V) = | B |$ .

This definition matches our intuition for finite-dimensional vector spaces very well, and it even works for a number of infinite-dimensional vector spaces too.

Example: $ℝ [x]$ is the vector space of polynomials

p (x) = α_{0} + α_{1} x + α_{2} x^{2} + α_{3} x^{3} + \dots + α_{d} x^{d} for some d \geq 0 and α_{i} \in ℝ .

The set of monomials ${1, x, x^{2}, x^{3}, \dots}$ is a basis of this vector space. It has countably many elements, so the dimension of $ℝ [x]$ is countably infinite.

However, when it comes to infinite-dimensional vector spaces we quickly run into cases where our intuition stops working.

Example: $C^{\infty} ([0, 1])$ is the set of all real-valued functions that are continuous and infinitely differentiable on the interval $[0, 1] \subset ℝ$ .

$C^{\infty} ([0, 1])$ forms an infinite-dimensional vector space over $ℝ$ using the usual function addition and scalar multiplication operations:

$(f + g) (x) = f (x) + g (x)$
$(α f) (x) = α f (x)$

The functions in $C^{\infty} ([0, 1])$ are about as well-behaved as functions get and yet there is no countable basis for this vector space: The set ${x \mapsto e^{λ x} | λ \in ℝ}$ is linearly independent and uncountable, so the dimension of $C^{\infty} ([0, 1])$ is at least $| ℝ |$ .

While countable sets of functions like polynomials can approximate smooth functions arbitrarily well, pure linear combinations only allow finite sums. Any finite combination of polynomials is just another polynomial of finite degree; it can never produce $e^{x}$ or $sin (x)$ .

But we can construct even simpler vector spaces with uncountably many dimensions.

Example: $ℝ$ is an infinite-dimensional vector space over $ℚ$ with vector addition and scalar multiplication defined using standard addition and multiplication. The set of vectors is the real numbers and the field over which this is a vector space is the rational numbers. Linear combinations are of the form

u = α_{1} v_{1} + α_{2} v_{2} + \dots + α_{n} v_{n} with α_{1}, \dots, α_{n} \in ℚ, v_{1}, \dots, v_{n} \in ℝ .

That is, the coefficients are rational numbers and the vectors are real numbers.

Let’s assume that the dimension of this vector space is finite. Then this vector space has a basis $B = {b_{1}, b_{2}, \dots, b_{n}} \subset ℝ$ and every element in $ℝ$ has a representation as a linear combination of this basis:

ℝ = {α_{1} b_{1} + α_{2} b_{2} + \dots + α_{n} b_{n} | α_{1}, \dots, α_{n} \in ℚ}

However, this would imply $| ℝ | = | ℚ^{n} |$ , but that is a contradiction: Since $ℚ$ is countable, the product $ℚ^{n}$ is also countable. However, $ℝ$ is uncountable, so $| ℝ | \neq | ℚ^{n} |$ .

Therefore, the dimension of this vector space cannot be finite. In fact, because any finite linear combination over a countable basis would still only generate a countable set of numbers, the dimension must be uncountably infinite.

This leads us directly to one central question that is very confusing: What does a basis of an infinite-dimensional vector space actually look like? Unfortunately, the answer to that is that when infinity rears its beautifully enormous head, constructive mathematics tends to leave the room.

The proof that all vector spaces have a basis requires the Axiom of Choice. It states that for any set of non-empty sets, there exists a choice function that maps each of the element sets to an element of that set. Whenever the Axiom of Choice is needed, it means we don’t know what that choice function is, which leaves us with no example of the object whose existence was proven.

In fact, if the Axiom of Choice is not assumed, but instead we assume that every vector space has a basis as an axiom, then it’s possible to prove the Axiom of Choice. That is, both are equivalent.

Coordinates

To see why our intuition fails so severely for these infinite-dimensional vector spaces, it helps to revisit why finite dimensions feel so intuitive in the first place: coordinates.

One closely related topic to vector space bases is vector space coordinates. If we have a finite basis $B = (b_{1}, b_{2}, \dots, b_{n})$ of a vector space $V$ over $𝔽$ , we can define a map $ϕ : 𝔽^{n} \to V$ with

ϕ (α_{1}, α_{2}, \dots, α_{n}) = α_{1} b_{1} + α_{2} b_{2} + \dots + α_{n} b_{n} .

This allows us to represent any finite-dimensional vector space in the usual way using coordinate tuples. In fact, this allows us to handle any finite-dimensional vector space over $ℝ$ like the standard vector spaces $ℝ^{n}$ .

Example: Let $B = (x \mapsto sin (x), x \mapsto cos (x))$ be the basis of a vector space of functions $V \subset C^{\infty} (ℝ)$ . By construction, this vector space has the dimension $dim (V) = 2$ .

With this basis, we can define a coordinate system $ϕ : ℝ^{2} \to V$ with

ϕ (α_{1}, α_{2}) = α_{1} sin + α_{2} cos

We can handle this vector space the same way as the well-known $ℝ^{2}$ vector space. This is actually very useful in practice, for example it can be used to visualize Quadrature Amplitude Modulation (QAM) using Constellation Diagrams.

Finite-dimensional function spaces like this also come up in solutions to homogeneous linear differential equations. In this case, the vector space $V$ is the space of all solutions to $f^{″} + f = 0$ .

Thinking of vector spaces as standard $ℝ^{n}$ coordinate tuples is natural because coordinates make finite-dimensional spaces easy to navigate. But this convenient representation is not what defines vector spaces. When a vector space has an infinite basis, the coordinate representation breaks down and our intuition for vector spaces stops working. We need to build a different kind of intuition.

Intuition

I titled this article Infinite-Dimensional Vectors, but there is no such thing: Dimensionality is not a property of a vector, but of the space it’s defined in. Consequently, a single vector from an infinite-dimensional vector space is something a human can comprehend. Furthermore, every vector in a vector space consists of a finite linear combination of basis vectors, even in infinite dimensions.

This means that every calculation on vectors is contained in a finite-dimensional subspace. Consider an infinite-dimensional vector space $V$ and two vectors $v, w \in V$ . Because $v$ and $w$ are made of a finite number of basis vectors, adding them is an operation confined to a finite-dimensional subspace. More generally, any finite set of vectors can only span a finite-dimensional subspace and every theorem from standard linear algebra applies there.

In other words, even infinite-dimensional vector spaces are structures we can reason about. We can even use our existing finite-dimensional intuition locally.

However, there’s a catch: Bases for infinite-dimensional vector spaces can be impossible to grasp. To make infinite-dimensional vector spaces truly useful, that problem needs to be addressed. This is what topology was invented for, but that’s a topic for a different article.

A general outline of the genesis of vector space theory

This is the basis theorem.

Existence of bases implies the axiom of choice