[API Proposal]: Tensor Operations Per Dimension

Background and motivation

We have exposed lots of functions we can use with Tensor<T>. These currently all operate on the entire backing memory of the Tensor. The problem with this is that many times you want to do the operation on whole dimensions, rather than the whole data, and we have no easy way of providing this currently. For a very simple example say your Tensor<T> is 2d, think of it like an Excel table. In Excel we can sum the whole table, but we can also sum each row, for example. This is the functionality we need to add to Tensor<T>.

The approach we have taken means we don't have to add overloads to every operation that could do this type of behavior. Currently we are planning on these methods needing this behavior.

CosineSimilarity
StdDev
Average
Max
MaxMagnitude
MaxMagnitudeNumber
MaxNumber
Min
MinMagnitude
MinMagnitudeNumber
MinNumber
Norm
Product
SoftMax
Sum
SumOfSquares
IndexOfMax
IndexOfMaxMagnitude
IndexOfMin
IndexOfMinMagnitude

API Proposal

namespace System.Numerics.Tensors;

public interface IReadOnlyTensor<TSelf, T> : IEnumerable<T>
    where TSelf : IReadOnlyTensor<TSelf, T>
{
    //used to easily get a sub dimension of a tensor. Like all the Rows of a tensor for example.
    TSelf SliceAlongDimension(int dimension, nint index);
}

public static partial class Tensor
{
        // These are for things like Sum that take in a tensor and return a single T.
    public delegate T TensorAggregationPredicate<T>(in ReadOnlyTensorSpan<T> tensor);
    public static Tensor<T> AggregateMany<T>(this Tensor<T> tensor, TensorAggregationPredicate<T> func, int dimension = -1);

    // These are for things like CosineSimilarity that take in a tensor and return a new tensor as well.
    public delegate Tensor<T> TensorSelectionPredicate<T>(in ReadOnlyTensorSpan<T> tensor);
    public static Tensor<T> SelectMany<T>(this Tensor<T> tensor, TensorSelectionPredicate<T> func, int dimension = -1);

    public static Tensor<T> ToTensor<T>(this IEnumerable<Tensor<T>> source);
}


public sealed class Tensor<T>
    : ITensor<Tensor<T>, T>
{
    public DimensionCollection Dimensions {get;}
    
    public sealed class DimensionCollection
        : System.Collections.Generic.ICollection<Tensor<T>>,
          System.Collections.Generic.IEnumerable<Tensor<T>>,
          System.Collections.Generic.IReadOnlyCollection<Tensor<T>>,
          System.Collections.ICollection
    {
        public int Count {get;}
        public void CopyTo(Tensor<T>[] array , int index)
        public Enumerator GetEnumerator();

        //All else explicitly implemented and match what Dictionary does   

        public struct Enumerator : IEnumerator<Tensor<T>>
        {
            
            internal Enumerator(Tensor<T> tensor, int dimension);

            public bool MoveNext();

            public void Reset();

            public void Dispose();

            Tensor<T> IEnumerator<Tensor<T>>.Current;

            object? IEnumerator.Current;
        }     
    }
}


public readonly ref struct ReadOnlyTensorSpan<T>
{
    public DimensionCollection Dimensions {get;}
    
    public ref struct DimensionCollection
        : System.Collections.Generic.ICollection<Tensor<T>>,
          System.Collections.Generic.IEnumerable<Tensor<T>>,
          System.Collections.Generic.IReadOnlyCollection<Tensor<T>>,
          System.Collections.ICollection
    {
        public int Count {get;}
        public void CopyTo(Tensor<T>[] array , int index)
        public Enumerator GetEnumerator();

        //All else explicitly implemented and match what Dictionary does   

        public ref struct Enumerator : IEnumerator<Tensor<T>>
        {
            
            internal Enumerator(Tensor<T> tensor, int dimension);

            public bool MoveNext();

            public ref readonly ReadOnlyTensorSpan<T> Current;
        }     
    }
}

public ref struct TensorSpan<T>
{
    public DimensionCollection Dimensions {get;}
    
    public ref struct DimensionCollection
        : System.Collections.Generic.ICollection<Tensor<T>>,
          System.Collections.Generic.IEnumerable<Tensor<T>>,
          System.Collections.Generic.IReadOnlyCollection<Tensor<T>>,
          System.Collections.ICollection
    {
        public int Count {get;}
        public void CopyTo(Tensor<T>[] array , int index)
        public Enumerator GetEnumerator();

        //All else explicitly implemented and match what Dictionary does   

        public ref struct Enumerator : IEnumerator<Tensor<T>>
        {
            
            internal Enumerator(Tensor<T> tensor, int dimension);

            public bool MoveNext();

            public ref TensorSpan<T> Current;
        }     
    }   
}

API Usage

Tensor<int> t1 = Tensor.Create([1,2,3,4,5,6,7,8,9], [3,3]);

// Will be [1,2,3]
Tensor<int> slice =  t1.SliceAlongDimension(0,0);

// Will be [7, 8, 9]
slice =  t1.SliceAlongDimension(0,2);
for (int i = 0; i < 3; i++)
{
    // Get each slice
    t1.SliceAlongDimension(0,i);
    // Can now do something which each sub slice.
}

Tensor<int> tensor = Tensor.Create([1, 2, 3, 4], [2, 2]);

// Sum tensor will be [3, 7]
Tensor<int> sum = t.AggregateMany(t => Sum(t), 0);

// CosineTensor would be [1,1,1]
Tensor<float> tensor1 = Create<float>([1, 2, 3, 4, 5, 6, 7, 8, 9], [3, 3], [], false);
Tensor<float> cosineTensor = tensor1.SelectMany(t => CosineSimilarity<T>(t, t), 0);

Alternative Designs

We could explicitly overload every method that has this behavior. That would mean an additional 20 overloads just for now. As time goes on this number would grow. If a user wanted to do this with a method we hadn't overloaded yet, they would either have to do it themselves (which we have had happen recently), or they would have to wait. Our current approach allows us to avoid both of these problems.

Risks

Low risk because it's a preview type and its all new features.

Looks great! Can you give examples from higher order tensors? For example, how would I softmax a 3d tensor across the 0th dimension such that the softmax works on 2d slices?

Just noticed:

public static Tensor<T> AggregateMany<T>(this Tensor<T> tensor, TensorAggregationPredicate<T> func, int dimension = -1);

The default dimension = -1 - what is the behavior?

And:

public interface IReadOnlyTensor<TSelf, T> : IEnumerable<T>
    where TSelf : IReadOnlyTensor<TSelf, T>
{
    //used to easily get a sub dimension of a tensor. Like all the Rows of a tensor for example.
    TSelf DimensionalSlice(int dimension, int index);
}

What is the behavior for this example:

Tensor<int> tensor = Tensor.Create([1, 2, 3, 4, 2, 4, 6, 8], [2, 2, 2]);  // equivalent to: [[[1, 2], [3, 4]], [[2, 4], [6, 8]]]

Tensor<int> slice0 = tensor.DimensionalSlice(0, 1); // What would be in here? [[2, 4], [6, 8]] ? 
Tensor<int> slice1 = tensor.DimensionalSlice(1, 2); // What would be in here? [3, 4] ? 
Tensor<int> slice2 = tensor.DimensionalSlice(2, 7); // What would be in here? [8] ?

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

Video

Tensor.Aggregate(Many) isn't performing the same accumulation-based logic as Enumerable.Aggregate, so it shouldn't use the same verb.
- The logic being performed by Tensor.AggregateMany here is really just Select, where the selector reduces dimensionality.
The "Many" here is doing the opposite of the meaning of Linq, so we should drop it.
The delegate types were renamed to TensorScalarSelector and TensorSelector to follow up with the renames of AggregateMany and SelectMany
SelectMany and AggregateMany (now both select) changed defaults from -1 to 0.

namespace System.Numerics.Tensors;

public partial interface IReadOnlyTensor<TSelf, T>
{
    TSelf SliceAlongDimension(int dimension, nint index);
}

public static partial class Tensor
{
    public delegate T TensorScalarSelector<T>(in ReadOnlyTensorSpan<T> tensor);
    public delegate Tensor<T> TensorSelector<T>(in ReadOnlyTensorSpan<T> tensor);

    public static Tensor<T> Select<T>(this Tensor<T> tensor, TensorScalarSelector<T> func, int dimension = 0);
    public static Tensor<T> Select<T>(this Tensor<T> tensor, TensorSelector<T> func, int dimension = 0);

    public static Tensor<T> ToTensor<T>(this IEnumerable<Tensor<T>> source);
}

public sealed class Tensor<T>
    : ITensor<Tensor<T>, T>
{
    public DimensionCollection Dimensions {get;}
    
    public sealed class DimensionCollection
        : System.Collections.Generic.ICollection<Tensor<T>>,
          System.Collections.Generic.IEnumerable<Tensor<T>>,
          System.Collections.Generic.IReadOnlyCollection<Tensor<T>>,
          System.Collections.ICollection
    {
        public int Count {get;}
        public void CopyTo(Tensor<T>[] array , int index);
        public Enumerator GetEnumerator();

        //All else explicitly implemented and match what Dictionary does   

        public struct Enumerator : IEnumerator<Tensor<T>>
        {
            internal Enumerator(Tensor<T> tensor, int dimension);
            public bool MoveNext();
            public void Reset();
            public void Dispose();
            Tensor<T> IEnumerator<Tensor<T>>.Current {get; }
            object? IEnumerator.Current { get; }
        }     
    }
}

public readonly ref struct ReadOnlyTensorSpan<T>
{
    public DimensionCollection Dimensions {get;}
    
    public ref struct DimensionCollection
        : System.Collections.Generic.ICollection<Tensor<T>>,
          System.Collections.Generic.IEnumerable<Tensor<T>>,
          System.Collections.Generic.IReadOnlyCollection<Tensor<T>>,
          System.Collections.ICollection
    {
        public int Count {get;}
        public void CopyTo(Tensor<T>[] array , int index);
        public Enumerator GetEnumerator();

        //All else explicitly implemented and match what Dictionary does   

        public ref struct Enumerator : IEnumerator<Tensor<T>>
        {
            internal Enumerator(Tensor<T> tensor, int dimension);
            public bool MoveNext();
            public ref readonly ReadOnlyTensorSpan<T> Current { get; }
        }     
    }
}

public ref struct TensorSpan<T>
{
    public DimensionCollection Dimensions {get;}
    
    public ref struct DimensionCollection
        : System.Collections.Generic.ICollection<Tensor<T>>,
          System.Collections.Generic.IEnumerable<Tensor<T>>,
          System.Collections.Generic.IReadOnlyCollection<Tensor<T>>,
          System.Collections.ICollection
    {
        public int Count {get;}
        public void CopyTo(Tensor<T>[] array , int index);
        public Enumerator GetEnumerator();

        //All else explicitly implemented and match what Dictionary does   

        public ref struct Enumerator : IEnumerator<Tensor<T>>
        {
            internal Enumerator(Tensor<T> tensor, int dimension);
            public bool MoveNext();
            public ref TensorSpan<T> Current { get; }
        }     
    }   
}

Regarding:

public partial interface IReadOnlyTensor<TSelf, T>
{
    TSelf SliceAlongDimension(int dimension, nint index);
}

After the API design review, I don't think that this is intuitive as I thought in the beginning.
In multidimensional cases, the index parameter isn't intuitive to me - the index would run higher than any given dimension in the tensor, and calculating it isn't trivial.

Given a 3-dimensional tensor, to enumerate with for loops on 2 dims and slice on the third would look like so:

Tensor<T> t  = Tensor.Create([1, 2, 3, 4, 5, 6, 7, 8], [2,2,2]);
nint maxIndex = t.Lengths[0] * t.Lengths[1];
for (nint i = 0; i < maxIndex; i++) 
{
      Tensor<T> slice = t.SliceAlongDimension(i);
      ...
}

Which isn't an intuitive way to loop over an "n dimensional array".

So more likely usage would look like so:

Tensor<T> t  = Tensor.Create([1, 2, 3, 4, 5, 6, 7, 8], [2,2,2]);
for (nint i = 0; i < t.Lengths[0]; i++) 
{
     for (nint j = 0; j < t.Lengths[1]; j++) 
     {
           // I actually got the indexing wrong on my first try
           Tensor<T> slice = t.SliceAlongDimension(i * t.Lengths[1] + j);
           ...
     }
}

From this usage it is unclear (as also discussed in the api review) what is the shape and content of the slice in the loop.

I would propose to change to instead pass a params ReadOnlySpan<nint> indices, with this API:

public Tensor<T> Index(params ReadOnlySpan<NRange> indices)

And then:

Tensor<T> t  = Tensor.Create([1, 2, 3, 4, 5, 6, 7, 8], [2,2,2]);
for (nint i = 0; i < t.Lengths[0]; i++) 
{
     for (nint j = 0; j < t.Lengths[1]; j++) 
     {
           Tensor<T> slice = t.Index(i, j);
           ...
     }
}

And then from my previous question:

Tensor tensor = Tensor.Create([1, 2, 3, 4, 2, 4, 6, 8], [2, 2, 2]); // equivalent to: [[[1, 2], [3, 4]], [[2, 4], [6, 8]]]

Tensor slice0 = tensor.DimensionalSlice(0, 1); // What would be in here? [[2, 4], [6, 8]] ?
Tensor slice1 = tensor.DimensionalSlice(1, 2); // What would be in here? [3, 4] ?
Tensor slice2 = tensor.DimensionalSlice(2, 7); // What would be in here? [8] ?

Would be:

Tensor<int> tensor = Tensor.Create([1, 2, 3, 4, 2, 4, 6, 8], [2, 2, 2]);  // equivalent to: [[[1, 2], [3, 4]], [[2, 4], [6, 8]]]

Tensor<int> slice0 = tensor.Index(1); //  [[2, 4], [6, 8]]
Tensor<int> slice1 = tensor.Index(0, 1); // [3, 4]
Tensor<int> slice2 = tensor.Index(1, 1, 1); // [8]

But this is just an indexer:

Tensor<int> tensor = Tensor.Create([1, 2, 3, 4, 2, 4, 6, 8], [2, 2, 2]);  // equivalent to: [[[1, 2], [3, 4]], [[2, 4], [6, 8]]]

Tensor<int> slice0 = tensor[1]; //  [[2, 4], [6, 8]]
Tensor<int> slice1 = tensor[0, 1]; // [3, 4]
Tensor<int> slice2 = tensor[1, 1, 1]; // [8]

But this doesn't work since:

public ref T this[params scoped ReadOnlySpan<nint> indexes]

Returns a scalar.
Can the index be changed to return a Tensor?
To enable getting a scalar back, I would add the following API instead:

public T Scalar(params ReadOnlySpan<nint> indexes);
public T Scalar();  // Throws if not a scalar tensor?

Regarding:

 public struct Enumerator : IEnumerator<Tensor<T>>
 {
     internal Enumerator(Tensor<T> tensor, int dimension);
     ...
 }

How will the dimension get passed to this Enumerator?
During the api review it was discussed that the usage should be:

foreach (Tensor<T> tensorSlice in tensor.Dimensions) 
{
    ...
}

But how can I set the dimension? And given my previous change to Slice - we would need to change it to ReadOnlySpan<nint>

@michaelgsharp

@michaelgsharp @tannergooding making sure you saw my comment on the design review

In multidimensional cases, the index parameter isn't intuitive to me - the index would run higher than any given dimension in the tensor, and calculating it isn't trivial.

dimension is [0, Rank). index is then [0, Product(Lengths[..(dimension + 1)])]

That is, consider a 2x3x4 tensor. This has a depth of 2, each of which has 3 rows, each of which has 4 columns. The entire tensor then has 6 total rows (3 rows to a depth of 2) and 24 total elements (4 columns across 6 total rows).

Thus the ranges are as follows, which allows you to iterate all 0-dimensional (elements), all 1-dimensional (rows), or all 2-dimensional tensors contained in the 3-dimensional tensor:

SliceAlongDimension(0, index);    // index is [0, 2), returns a tensor (3x4 elements)
SliceAlongDimension(1, index);    // index is [0, 6), returns a row (4 elements)
SliceAlongDimension(2, index);    // index is [0, 24), returns a scalar (1 element)

Given a 3-dimensional tensor, to enumerate with for loops on 2 dims and slice on the third would look like so:

You'd normally just get the Count via Dimensions[dimension].Count which tells you how many total entries exist for that dimension (2, 6, 24, etc)

I would propose to change to instead pass a params ReadOnlySpan indices, with this API:

This makes indexing much more expensive and is already possible using Slice itself or the indexer that takes a range. The purpose of SliceAlongDimension is to make it cheaper and more efficient to iterate a particular dimensional slice and therefore to skip the normal slicing considerations that might exist.

Can the index be changed to return a Tensor?

No. This would defeat the typical expectation of an indexer. There is a separate indexer that takes a range and where you can do tensor[0.., .., ..] to get the "equivalent" of SliceAlongDimension(0, 0).

During the api review it was discussed that the usage should be:

Not quite. It should rather be the following, which specifies you want to iterate all rows in the 2x3x4:

foreach (var slice in tensor.Dimensions[1])
{
    ...
}

That is, it is equivalent to:

for (int i = 0; i < tensor.Dimensions[1].Count; i++)
{
    var slice = tensor.SliceAlongDimension(1, i);
    ...
}

@tannergooding

Makes sense @tannergooding .
I don't see an indexer defined on DimensionCollection type. Is that a mistake?

Navigation Menu

Repository navigation

[API Proposal]: Tensor Operations Per Dimension #113068

Background and motivation

API Proposal

API Usage

Alternative Designs

Risks

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[API Proposal]: Tensor Operations Per Dimension #113068

Description

Background and motivation

API Proposal

API Usage

Alternative Designs

Risks

Activity

tjwald commented on Mar 3, 2025

tjwald commented on Mar 3, 2025

dotnet-policy-service commented on Mar 3, 2025

bartonjs commented on Mar 4, 2025

tjwald commented on Mar 5, 2025

tjwald commented on Mar 18, 2025

tannergooding commented on Mar 24, 2025

tjwald commented on Mar 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions