Repository navigation
Add fast image processor for ViViT - #40209
msminhas93 wants to merge 2 commits into
Conversation
|
Hi @yonigozlan! Could you please take a look. Thank you. |
yonigozlan
left a comment
yonigozlan
left a comment
There was a problem hiding this comment.
Hi @msminhas93, thanks for contributing, a few things can be simplified!
| def make_batched(videos) -> list[list]: | ||
| """Make videos batched for processing.""" | ||
| if isinstance(videos, (list, tuple)) and isinstance(videos[0], (list, tuple)): | ||
| return videos | ||
| elif isinstance(videos, (list, tuple)): | ||
| return [videos] | ||
| else: | ||
| return [[videos]] | ||
|
|
There was a problem hiding this comment.
We have make_nested_list_of_images in image_utils for that
| processed_videos = [] | ||
| for video in images: # images is a list of videos, each video is a list of frames | ||
| # Process each frame in the video | ||
| processed_frames = [] | ||
| for frame in video: | ||
| # Apply transformations to individual frame | ||
| if do_resize: | ||
| frame = self.resize(image=frame, size=size, interpolation=interpolation) | ||
|
|
||
| if do_center_crop: | ||
| frame = self.center_crop(frame, crop_size) | ||
|
|
||
| # Handle rescaling with offset parameter | ||
| if do_rescale: | ||
| frame = self.rescale(frame, rescale_factor, offset=offset) | ||
|
|
||
| # Handle normalization | ||
| if do_normalize: | ||
| frame = self.normalize(frame, image_mean, image_std) | ||
|
|
||
| processed_frames.append(frame) | ||
|
|
||
| # Stack frames for this video | ||
| if return_tensors: | ||
| video_tensor = torch.stack(processed_frames, dim=0) | ||
| processed_videos.append(video_tensor) | ||
| else: | ||
| processed_videos.append(processed_frames) | ||
|
|
||
| # Stack videos if returning tensors | ||
| if return_tensors: | ||
| processed_videos = torch.stack(processed_videos, dim=0) | ||
|
|
||
| return BatchFeature(data={"pixel_values": processed_videos}, tensor_type=return_tensors) |
There was a problem hiding this comment.
You can process fully by batch still. just use the code from the parent class, and set is_nested=True in group_images_by_shape and reorder_images
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, vivit |
|
Thank you for the comments! I've updated my PR @yonigozlan |
|
@yonigozlan If you could please look at the PR that'll be awesome Thank you! |
|
Thanks for the PR, and sorry it didn't make it in! This branch is based on v4, and v5 is very different, so rather than rebasing we'd love to invite you to open a new PR against
Closing this one in the meantime, thanks again! |
What does this PR do?
This PR implements a fast PyTorch image processor for ViViT (
VivitImageProcessorFast) that extends the existingBaseImageProcessorFastclass. The new processor provides optimized image processing capabilities specifically designed for video understanding tasks using the ViViT model.Key Features
Before submitting
Who can review?
@yonigozlan