Repository navigation
feat(people-and-planet-ai): modernize wildlife image classification with Gemini 3.8 Flash, Gemini Embedding 2, and Apache Beam RunInference - #14701
Open
iennae wants to merge 19 commits into
Conversation
…ith Gemini and Apache Beam RunInference Problem: The legacy people-and-planet-ai/image-classification sample relied on deprecated AutoML Vision APIs (train_model.py, deploy_model.py, predict.py), pointed to a defunct Azure Blob Storage URL (HTTP 403) for the LILA WCS Camera Traps dataset, required copying gigabytes of camera trap images into a user bucket, excluded rare species with fewer than 50 training images, and required hours of model training plus an always-on endpoint. Solution: - Update create_images_metadata_table.py (region tag dataflow_wildlife_ingest_metadata) to ingest and clean WCS camera trap annotations from LILA Science's public Cloud Storage archive (https://storage.googleapis.com/public-datasets-lila/wcs/wcs_camera_traps.json.zip) into BigQuery. - Add classify_wildlife_image.py (region tag genai_wildlife_classify_image) using Gemini 3.8 Flash (gemini-3.8-flash) on the global endpoint with Pydantic Structured Outputs (WildlifeObservation) and habitat geo-fencing (habitat_region) to classify camera trap images in-place from gs://public-datasets-lila/ into species, taxonomic family, common name, animal count, juvenile presence, and empty/misfire flags. - Add embed_wildlife_image.py (region tag genai_wildlife_embed_image) using Gemini Embedding 2 (gemini-embedding-2) to generate 768-dimensional multimodal image embeddings for 1-shot rare species similarity search with BigQuery VECTOR_SEARCH. - Add run_wildlife_inference.py (region tag dataflow_wildlife_run_inference) using Apache Beam Sample.FixedSizePerKey and RunInference (GeminiWildlifeModelHandler) to run class-balanced batch inference directly against gs://public-datasets-lila/wcs-unzipped/ and write structured observations to BigQuery. - Remove deprecated train_model.py, deploy_model.py, and predict.py. - Update e2e_test.py, requirements.txt, requirements-test.txt, README.ipynb, README.md, and people-and-planet-ai/README.md for DevSite style compliance and end-to-end test coverage.
…ook setup/cleanup cells, and cite verified WCS counts Problem: README.ipynb replaced the original Apache Software Foundation (ASF) license cell header, lacked clarity on execution vs. setup time, required navigating to external console pages to enable APIs and create/clean up resources, and used approximate species skew wording. Solution: - Restore the original ASF Apache 2.0 header in Cell 0 of README.ipynb while keeping 2026 copyright headers on the new/refactored Python files. - Clarify the time estimate (~25-30 minutes total: ~10 minutes Dataflow execution plus ~15-20 minutes setup and exploration) and remove the 'free if you use Cloud credits' parenthetical. - Add executable cells in README.ipynb to enable the required APIs, create the regional Cloud Storage bucket and BigQuery dataset, and clean up the BigQuery dataset and Cloud Storage staging files directly in the notebook. - Update the species distribution section with verified counts from wcs_camera_traps.json.zip (736,952 valid annotations across 639 species; 95,788 tayassu pecari images vs. 54 dicerorhinus sumatrensis and 1 tolypeutes matacus; 359 of 639 species having fewer than 50 images).
… notebook_template.ipynb Problem: README.ipynb used custom legacy headings and project configuration variables instead of the standard GoogleCloudPlatform/vertex-ai-samples notebook_template.ipynb layout. Solution: - Adopt the vertex-ai-samples notebook_template.ipynb header and section hierarchy (# Title + 4-way launch table, ## Overview, ## Get started, ### Install required packages, ### Import libraries, ### Authenticate your notebook environment (Colab only), ### Set Google Cloud project information, ### Initialize Gen AI SDK, tutorial sections, and ## Cleaning up). - Standardize notebook configuration parameters on PROJECT_ID, LOCATION, BUCKET_NAME, CLOUD_STORAGE_DIRECTORY, BIGQUERY_DATASET, BIGQUERY_TABLE, and PREDICTIONS_TABLE across all pipeline, interactive inference, embedding, and cleanup cells.
…ll 0 of README.ipynb Problem: Aligning README.ipynb with notebook_template.ipynb replaced the original Apache Software Foundation (ASF) license notice in Cell 0. Solution: Restore the original Licensed to the Apache Software Foundation (ASF) notice in Cell 0 of README.ipynb.
…CT_ID and add Colab Enterprise VPC note Problem: Cell 13 rendered @PARAM form fields asking the user to paste in a bucket name and dataset name, and users opening Colab Enterprise in organizations with constraints/compute.skipDefaultNetworkCreation encountered a blank Network dropdown and read-only preview mode. Solution: - Keep only PROJECT_ID and LOCATION in the Set Google Cloud project information @PARAM cell. - Add a dedicated Create a Cloud Storage bucket and BigQuery dataset subsection that automatically derives BUCKET_NAME = f'{PROJECT_ID}-wildlife-insights' and BIGQUERY_DATASET = 'samples', enables the required APIs, and creates the bucket and dataset inline. - Add a collapsible note under the launch buttons explaining how to create the default VPC network and regional subnet if the Network dropdown is blank when importing into Colab Enterprise.
…tachedHead notice
…mage-classification
product-auto-label
Bot
added
samples
|
Here is the summary of changes. You are about to add 4 region tags.
This comment is generated by snippet-bot.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request modernizes the wildlife image classification sample by transitioning from legacy AutoML Vision to Gemini-based solutions. It introduces scripts for multimodal classification using gemini-3.8-flash with optional habitat geo-fencing, generating vector embeddings with gemini-embedding-2 for similarity search, and running class-balanced batch inference using Apache Beam on Dataflow. The Jupyter notebook and documentation are updated to reflect this serverless, lower-cost architecture. The reviewer feedback focuses on enhancing pipeline robustness and error handling, including implementing retries for external HTTP downloads, wrapping individual element processing in try-except blocks to prevent pipeline crashes, catching general exceptions instead of specific GCP client exceptions during Beam pipeline execution, and safely handling the embedding response attributes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Modernizes the People & Planet AI: Wildlife Insights (
people-and-planet-ai/image-classification) sample and interactive notebook by replacingdeprecated AutoML Vision training and dedicated endpoint deployment with Gemini 3.8 Flash (
gemini-3.8-flash), Gemini Embedding 2(
gemini-embedding-2), Apache BeamRunInferenceon Dataflow, and BigQueryVECTOR_SEARCH.DO NOT MERGE. Once reviewed, I need to modify some sections around the checkout rather than checking out from this branch. That's a small change, but I wanted to make it possible to validate the notebook without having to have reviewers modify the notebook.
I have tested this in Colab Enterprise.
Why this matters for conservation workloads
In the LILA Science Wildlife Conservation Society (WCS) Camera Traps dataset (
736,952valid annotations across639species), species counts areheavily skewed—from
95,788photos of common species (tayassu pecari) down to54photos of critically endangered Sumatran rhinoceros(
dicerorhinus sumatrensis) and1photo of southern three-banded armadillo (tolypeutes matacus).min_images_per_class = 50to train a supervised classifier, excluding359of639species (56%)—including the rarest and most endangered animals. Zero-shot classification withgemini-3.8-flashsupportsmin_images_per_class = 1while keepingSample.FixedSizePerKey(max_images_per_class)in Apache Beam so common species do not dominate batchinference budgets.
build an AutoML CSV import file, Gemini reads images directly in place from
gs://public-datasets-lila/wcs-unzipped/.WildlifeObservation), each inference returnsspecies_scientific_name,taxon_family(for family-level rollups on blurry nighttime frames),common_name,animal_count,juvenile_present(totrack breeding and population recovery),
is_empty_or_misfire, andconfidence, with optionalhabitat_regiongeo-fencing.gemini-embedding-2(768-dimensional multimodal embeddings) and BigQueryVECTOR_SEARCHto findvisually similar camera trap frames from a single reference sighting.
RunInference.Key Changes
1. Standalone DevSite Code Snippets
create_images_metadata_table.py([START dataflow_wildlife_ingest_metadata]): Apache Beam pipeline that streams and cleans the LILA WCS CameraTraps metadata archive (
gs://public-datasets-lila/wcs/wcs_camera_traps.json.zip) into BigQuery.classify_wildlife_image.py([START genai_wildlife_classify_image]): Classifies camera trap imagery in-place from Cloud Storage usinggemini-3.8-flashon the global endpoint with structured output (WildlifeObservation) and optionalhabitat_regiongeo-fencing.embed_wildlife_image.py([START genai_wildlife_embed_image]): Generates 768-dimensional multimodal vector embeddings for camera trap imageryusing
gemini-embedding-2on the global endpoint.run_wildlife_inference.py([START dataflow_wildlife_run_inference]): Runs class-balanced per-species sampling (Sample.FixedSizePerKey) andbatch Gemini inference using Apache Beam
RunInference(GeminiWildlifeModelHandler), writing structured ecological observations to BigQuery.train_model.py,deploy_model.py, andpredict.py.2. Notebook & Documentation (
README.ipynb,README.md)README.ipynbwith the standard Google Cloud notebook layout (4-badge launch bar for Colab, Colab Enterprise, Workbench, and GitHub; costcalculator disclosure; guarded Colab authentication;
example-project-id@paramdefaults withos.environfallbacks; andIS_TESTINGcleanupguard).
git cloneand kernel restart steps with an isolatedgit sparse-checkoutand in-place%pip install.--soft-delete-duration=0s) on the temporary Dataflow staging bucket to prevent retention warnings and unnecessary storagecharges.
VECTOR_SEARCHsection overgemini-embedding-2camera trap embeddings.Post-Approval Actions