Repository navigation
problem when use Float32Array in object's property #76
Description
Problem
Problem happen when use Float32Array in object's property.
indicate minimum jest test below
import { encode, decode } from "@msgpack/msgpack";
test("issue", () => {
const f32 = new Float32Array([1.1, 1.2, 1.3]);
const uint8 = new Uint8Array(f32.buffer);
expect(new Float32Array(uint8.buffer)).toEqual(f32); //success
const data = encode({ audio: uint8 });
const { audio } = decode(data) as { audio: Uint8Array };
expect(uint8).toEqual(audio); //success
expect(uint8.buffer).toEqual(audio.buffer); //success
const fail = new Float32Array(audio.buffer);
expect(fail).toEqual(f32); //fail
const success = new Float32Array(new Uint8Array(Object.values(audio)).buffer);
expect(success).toEqual(f32); //success
});this test fail in
const fail = new Float32Array(audio.buffer);
expect(fail).toEqual(f32); //failEnviroment
Node.js : 11.15.0
TypeScript : 3.5.2
Test Framework : Jest
Activity
gfx commentedon Jul 11, 2019
This is the expected behavior of typed arrays.
First of all, the way to chang a typed array class is not
new Float32Array(u8array.buffer), but you have to provide the range (offset, length) of the buffer:This is because TypedArray's
bufferis a pointer to a slice of an ArrayBuffer. Thus,u8arrayandf32arrayshares the samebuffer: ArrayBuffer.crabmusket commentedon Sep 19, 2021
I have been doing some work on an extension that handles TypedArrays natively and also aligns them at encoding time, rather than doing a copy when decoding.
Here's a readme describing how my extension works. And in this example you can see the encoding/decoding of different TypedArrays with no boilerplate.
However, in order to align during encoding I had to implement my own "plugin" system which works slightly differently to the existing extensions. This happens here. Basically the "plugin" just calls the encoder's private methods, instead of returning a blob which is then encoded as an
ext.This means my code has to write the ext header itself, but it makes the whole process more efficient and straightforward.
If I'd used a regular extension, it would require some tricky logic to "guess" which type of ext the encoder was going to use for the typed array data and adjust the alignment accordingly. It would also involve an intermediate copy (from TypedArray to ExtData, then from ExtData to the encoding buffer).
I'm not sure if this is of interest to you @gfx - I ask because I'm wondering whether something like my plugin system would ever make it into upstream, or whether I should maintain a fork or try harder to work with the existing extension system. I'm going to experiment with using my fork in some upcoming work and see how I feel about it. My use cases involve pushing around large arrays of float data, obviously!
I don't know whether it seems like I'm trying way too hard to optimise things, or if the idea of aligning data within the payload is a portability concern. Obviously this is really focused on JS-to-JS use cases, but other languages could read the memory into typed array primitives too I'm sure, and get the performance boost of alignment-without-copying.
EDIT: I've realised I can probably make the "really low level extension" concept fit more neatly into the existing code. I might give that a go then propose a separate issue/PR.
EDIT 2: updated my fork to have less intrusive extension changes. ExtData now has a write method and my plugin returns a subclass which does the actual encoding. If I were designing this properly, I'd replace the use of
ExtDatawith an interface to avoid mandatory inheritance hierarchies.gfx commentedon Sep 21, 2021
@crabmusket Thanks for the interesting report. Can you file an independent issue? Yours is definitely not related to the original issue.
crabmusket commentedon Sep 21, 2021
@gfx done!