Skip to content

Add DataFrame.fromArrow - #250

Merged
bryevdv merged 4 commits into
rapidsai:mainfrom
bryevdv:bryanv/arrow_ipc
Jun 29, 2021
Merged

bryevdv merged 4 commits into
rapidsai:mainfrom
bryevdv:bryanv/arrow_ipc

Conversation

@bryevdv

@bryevdv bryevdv commented Jun 26, 2021 •

Copy link
Copy Markdown
Contributor

Add a static DataFrame.fromArrow method that can adapt Arrow tables stored in-memory in Arrow IPC format.

This is still WIP but early comments welcome. Some notes:

  • As implemented, this requires a copy. Plan to to push improvements upstream to cudf::from_arrow to allow it to perform zero-copy when feasible (e.g. ipc buffers)
  • Tests are a bit thin. Open to suggestions to improve coverage parsimoniously
  • Still need to add an input type to fromArrow (similar to readCSV) so that full types can be deduced

@matekdev matekdev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial comments

Comment thread modules/cudf/src/data_frame.ts Outdated
Comment thread modules/cudf/src/table.ts Outdated
Comment thread modules/cudf/src/table/arrow.cpp
@bryevdv
bryevdv marked this pull request as ready for review June 28, 2021 18:12
@matekdev

Copy link
Copy Markdown
Contributor

Reviewed in the morning and didn't realize I accidently approved. Will give it another look over.

@matekdev

Copy link
Copy Markdown
Contributor

BINDINGS.md needs to be updated, otherwise, got nothing else to comment Might be good to get another review 👍

@bryevdv

bryevdv commented Jun 29, 2021

Copy link
Copy Markdown
Contributor Author

Good catch, updated!

@bryevdv
bryevdv merged commit 2a58316 into rapidsai:main Jun 29, 2021
@bryevdv
bryevdv deleted the bryanv/arrow_ipc branch June 29, 2021 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants