Vectors are coordinates with assumptions
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Compute dot products, norms, and cosine similarity.
- Explain how scaling changes a similarity ranking.
- Identify degenerate vectors and incompatible spaces.
Coordinates need a shared meaning
A vector is an ordered collection of numbers. Its usefulness comes from what the coordinates represent and which operations preserve that meaning. A three-coordinate product vector could store price, weight, and demand. An embedding might have hundreds of learned coordinates with no simple human label. In either case, comparing vectors assumes that corresponding coordinates belong to the same space.
Two embedding models that both output 768 numbers do not necessarily produce compatible spaces. Equal dimensionality is a shape check, not a semantic guarantee. Record the model and preprocessing version with stored vectors. If a query uses a new model while documents retain old vectors, a numerical similarity calculation can succeed while its ranking becomes meaningless. This is an example of a contract that extends beyond types.
Dot products combine alignment and size
For a = [1, 2] and b = [3, 4], the dot product is 1 times 3 plus 2 times 4, which equals 11. The Euclidean norm of a is the square root of 5. A dot product is large when vectors align and have large magnitudes, so it can reward both direction and scale. That may be desirable when magnitude intentionally encodes confidence or popularity.
To see the tradeoff, compare a query [1, 0] with candidates [2, 0] and [100, 100]. Their dot products are 2 and 100. The second wins despite pointing diagonally, because its magnitude overwhelms the direction difference. Whether that is correct depends on how the representation was trained and what the retrieval system is trying to rank.
Cosine removes one kind of scale
Cosine similarity divides the dot product by the product of the norms. For nonzero vectors, it measures directional alignment. The candidates above have cosine similarities 1 and approximately 0.707 with the query. If you normalize every nonzero vector to length one, dot product and cosine produce the same ranking. Normalization is a modeling choice; it should match the intended similarity metric.
A zero vector has no direction, so its cosine similarity is undefined. Returning zero can be a deliberate fallback, but silently doing so confuses absence of a representation with genuine orthogonality. The example raises an error for zero vectors. A production retrieval pipeline could instead exclude them and record a validation failure, making the missing representation visible.
Matrices are batches of linear combinations
A matrix can represent several output coordinates computed from the same input. If x = [2, 1] and the rows of W are [1, 0] and [1, -1], then Wx = [2, 1]. The first output copies the first coordinate; the second measures a difference. Neural network projections use the same operation with learned weights and many more coordinates.
Keep shapes explicit before optimizing code. With an input vector of length d and a matrix with k rows of length d, the result has length k. In a batch, add a leading dimension for examples. A transposed matrix can sometimes produce a plausible shape on square examples, so include rectangular test cases. Good shape reasoning prevents errors that a single successful execution would miss.
Feature scaling is part of the model
Suppose one coordinate is age in years and another is income in dollars. A raw Euclidean distance can be dominated by income simply because its numeric scale is larger. Standardizing features using training-set statistics can make the coordinates more comparable, but it changes the geometry. Applying a scaler fitted on the test set leaks information from future evaluation data into your model-building process.
The code contrasts magnitude-sensitive and direction-sensitive rankings with tiny vectors so every result can be checked by hand. It is not a recommendation to replace a trained embedding system with these coordinates. Use the exercise to develop a diagnostic habit: when a ranking looks strange, inspect coordinate meaning, normalization, model identity, and distance definition before assuming the search index is broken.
Work through the code
The example shows how a large vector wins under dot product but loses under cosine. The final matrix-vector multiplication computes two explicit features. Dimensions and zero norms are validated. Replace vectors with other hand-checkable coordinates before scaling up to array libraries.
import math
def dot(a, b):
if len(a) != len(b) or not a:
raise ValueError("nonempty matching dimensions required")
return sum(x * y for x, y in zip(a, b))
def cosine(a, b):
numerator = dot(a, b)
denominator = math.sqrt(dot(a, a) * dot(b, b))
if denominator == 0:
raise ValueError("zero vector has no direction")
return numerator / denominator
query = [1, 0]
candidates = {"aligned": [2, 0], "large": [100, 100]}
for name, vector in candidates.items():
print(f"{name}: dot={dot(query, vector):.1f} cosine={cosine(query, vector):.3f}")
weights = [[1, 0], [1, -1]]
print("projection:", [dot(row, [2, 1]) for row in weights])
aligned: dot=2.0 cosine=1.000 large: dot=100.0 cosine=0.707 projection: [2, 1]
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A search service normalizes documents but leaves queries unnormalized. For a fixed nonzero query, will dot-product ranking equal cosine ranking? Will the raw scores be directly comparable across queries?
Check your understanding
Two embedding models produce vectors of the same length. Can you safely mix their document and query vectors?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.