Skip to content

Pruning models from tf.keras.applications (efficientnet, resnet etc) #656

Open
Open
Pruning models from tf.keras.applications (efficientnet, resnet etc)#656
Assignees
liyunlu0618

Description

@ameyparanjape
ameyparanjape
opened on Apr 5, 2021

I am finetuning the tf.keras.applications model on my data for the problem of image classification. I was able to complete training and test results look good. In order to deploy the model, I wanted to perform some pruning and optimization. I tried following the Keras Pruning tutorial, but I could not figure out how to enable pruning on the efficientnet layer. If we could have a tutorial on this or a worked out example, that would be great, thank you!

Here is the model summary:

Model: "model_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_8 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
efficientnetb0 (Functional)  (None, 7, 7, 1280)        4049571   
_________________________________________________________________
global_average_pooling2d_3 ( (None, 1280)              0         
_________________________________________________________________
dense_6 (Dense)              (None, 8)                 10248     
_________________________________________________________________
dropout_3 (Dropout)          (None, 8)                 0         
_________________________________________________________________
dense_7 (Dense)              (None, 1)                 9         
=================================================================
Total params: 4,059,828
Trainable params: 10,257
Non-trainable params: 4,049,571

Activity

  1. Xhark
    assigned
    liyunlu0618
    on Apr 6, 2021

  2. Xhark commented on Apr 6, 2021

    @Xhark
    Xhark
    on Apr 6, 2021
    Member

    Hi liyunlu0618@, Would you please take a look?

  3. liyunlu0618 commented on Apr 6, 2021

    @liyunlu0618
    liyunlu0618
    on Apr 6, 2021
    Contributor

    Can you follow the example of pruning the whole model and try prune_low_magnitude(your model)? Then post the model summary or let us know if you run into any issues. Thanks!

  4. ameyparanjape commented on Apr 6, 2021

    @ameyparanjape
    ameyparanjape
    on Apr 6, 2021
    Author

    Thank you @liyunlu0618 , I have tried following the tutorial, here is the model.summary() and the error I get on using prune_low_magnitude() on my model:
    model.summary()

    Model: "model_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_8 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
efficientnetb0 (Functional)  (None, 7, 7, 1280)        4049571   
_________________________________________________________________
global_average_pooling2d_3 ( (None, 1280)              0         
_________________________________________________________________
dense_6 (Dense)              (None, 8)                 10248     
_________________________________________________________________
dropout_3 (Dropout)          (None, 8)                 0         
_________________________________________________________________
dense_7 (Dense)              (None, 1)                 9         
=================================================================
Total params: 4,059,828
Trainable params: 10,257
Non-trainable params: 4,049,571

    Error
    ValueError: Please initialize 'Prune' with a supported layer. Layers should either be a 'PrunableLayer' instance, or should be supported by the PruneRegistry. You passed: <class 'tensorflow.python.keras.engine.functional.Functional'>

  5. liyunlu0618 commented on Apr 7, 2021

    @liyunlu0618
    liyunlu0618
    on Apr 7, 2021
    Contributor

    Thanks for reporting the error. I'm working on a change to support pruning a model recursively. Will let you know once it's checked-in.

  6. ameyparanjape commented on Apr 7, 2021

    @ameyparanjape
    ameyparanjape
    on Apr 7, 2021
    Author

    Thank you @liyunlu0618

  7. liyunlu0618 commented on Apr 7, 2021

    @liyunlu0618
    liyunlu0618
    on Apr 7, 2021
    Contributor

    The PR is merged.
    #658

    You can try later with a nightly build and let us know if it works. Thanks!

  8. ameyparanjape commented on Apr 7, 2021

    @ameyparanjape
    ameyparanjape
    on Apr 7, 2021
    Author

    Thank you @liyunlu0618 , I am trying to get the tf-nightly installed and running. (never used tf-nightly before, always used stable)
    Here is what I am doing:

    1. Uninstall current stable tensorflow-2.4.1 (pip uninstall tensorflow)
    2. Install tf-nightly (tf_nightly-2.6.0.dev20210407) (pip install -U tf-nightly --user)

    (Please advise if I am not on the right track!)

    I got an error, I'm not sure if I am missing any additional step to get the installation right.
    Here is what my output/error log looks like:

    INFO:tensorflow:Enabling eager execution
INFO:tensorflow:Enabling v2 tensorshape
INFO:tensorflow:Enabling resource variables
INFO:tensorflow:Enabling tensor equality
INFO:tensorflow:Enabling control flow v2

---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-3-53c627d25dc5> in <module>
      1 # Import packages and set TPU/GPU environment if enabled
      2 
----> 3 import tensorflow as tf
      4 
      5 

~/.local/lib/python3.7/site-packages/tensorflow/__init__.py in <module>
    443     _main_dir = _os.path.join(_s, 'tensorflow/core/kernels')
    444     if _os.path.exists(_main_dir):
--> 445       _ll.load_library(_main_dir)
    446 
    447     # Load third party dynamic kernels.

~/.local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py in load_library(library_location)
    152 
    153     for lib in kernel_libraries:
--> 154       py_tf.TF_LoadLibrary(lib)
    155 
    156   else:

NotFoundError: /opt/conda/lib/python3.7/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so: undefined symbol: _ZN10tensorflow6thread10ThreadPool26TransformRangeConcurrentlyExxRKSt8functionIFvxxEE

  9. ameyparanjape commented on Apr 8, 2021

    @ameyparanjape
    ameyparanjape
    on Apr 8, 2021
    Author

    @liyunlu0618 Not sure how exactly, but now the tensorflow-nightly seems to be installed correctly. (I made sure by printing the version of tensorflow)
    I am still getting the same error. It seems like there might be difference between the <class 'tensorflow.python.keras.engine.functional.Functional'> objects and Recursive sequential layers object(I peeked into the PR). Also I am wondering that, since Pruning is a part of tensorflow_model_optimization, do I have to take any additional steps to update that to nightly as well?

    Here is my code snippet and error log for reference. It is unchanged from my last comment:

    import tensorflow_model_optimization as tfmot
print("Tensorflow_version : ",tf.__version__)
prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude
# Compute end step to finish pruning after 2 epochs.
batch_size = 128
epochs = 2
end_step = np.ceil(TRAIN_IMG_COUNT / batch_size).astype(np.int32) * epochs

# Define model for pruning.
pruning_params = {
      'pruning_schedule': tfmot.sparsity.keras.PolynomialDecay(initial_sparsity=0.50,
                                                               final_sparsity=0.80,
                                                               begin_step=0,
                                                               end_step=end_step)
}
model_for_pruning = prune_low_magnitude(model, **pruning_params)
# `prune_low_magnitude` requires a recompile.
model_for_pruning.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy,
              metrics=['accuracy'])

    Output Log (clipped):

    Tensorflow_version :  2.6.0-dev20210407
ValueError                                Traceback (most recent call last)
<ipython-input-16-cf5cd8efdcbd> in <module>
     23 
     24 
---> 25 model_for_pruning = prune_low_magnitude(model, **pruning_params)
     26 
     27 # `prune_low_magnitude` requires a recompile.
.
.
(Trimmed)
.
.
ValueError: Please initialize `Prune` with a supported layer. Layers should either be a `PrunableLayer` instance, or should be supported by the PruneRegistry. You passed: <class 'tensorflow.python.keras.engine.functional.Functional'>

  10. liyunlu0618 commented on Apr 8, 2021

    @liyunlu0618
    liyunlu0618
    on Apr 8, 2021
    Contributor

    Can you try a nightly build of tf-mot?

  11. ameyparanjape commented on Apr 8, 2021

    @ameyparanjape
    ameyparanjape
    on Apr 8, 2021
    Author

    Do you mind addressing how to use/install tf-mot nightly? I could not find any documentation on that.

    Currently I have this from pip freeze:

    tf-nightly==2.6.0.dev20210408
tensorflow-model-optimization==0.5.0

  12. liyunlu0618 commented on Apr 20, 2021

    @liyunlu0618
    liyunlu0618
    on Apr 20, 2021
    Contributor

    You may have to build from source at the moment:
    https://www.tensorflow.org/model_optimization/guide/install#installing_from_source

    Will let you know once we have a new release.

  13. Sai-222777 commented on Aug 17, 2025

    @Sai-222777
    Sai-222777
    on Aug 17, 2025

    Hi, is this issue resolved? I tried pruning EfficientNetB0, and I get the error that pruning of 'Functional' layer is not supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions