How to apply masking layer to sequential CNN model in Keras?

Multi tool use
Multi tool use












0















I have a problem of applying masking layer to CNNs in RNN/LSTM model.



My data is not original image, but I converted into a shape of (16, 34, 4)(channels_first). The data is sequential, and the longest step length is 22. So for invariant way, I set the timestep as 22. Since it may be shorter than 22 steps, I fill others with np.zeros. However, for 0 padding data, it's about half among all dataset, so with 0 paddings, the training cannot reach a very good result with so much useless data. Then I want to add a mask to cancel these 0 padding data.



Here is my code.



mask = np.zeros((16,34,4), dtype = np.int8)  
input_shape = (22, 16, 34, 4)
model = Sequential()
model.add(TimeDistributed(Masking(mask_value=mask), input_shape=input_shape, name = 'mask'))
model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name = 'conv1'))
model.add(TimeDistributed(BatchNormalization(), name = 'bn1'))
model.add(Dropout(0.5, name = 'drop1'))
model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv2'))
model.add(TimeDistributed(BatchNormalization(), name = 'bn2'))
model.add(Dropout(0.5, name = 'drop2'))
model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv3'))
model.add(TimeDistributed(BatchNormalization(), name = 'bn3'))
model.add(Dropout(0.5, name = 'drop3'))
model.add(TimeDistributed(Flatten(), name = 'flatten'))
model.add(GRU(256, activation='tanh', return_sequences=True, name = 'gru'))
model.add(Dropout(0.4, name = 'drop_gru'))
model.add(Dense(35, activation = 'softmax', name = 'softmax'))
model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['acc'])


Here's the model structure.

model.summary():



_________________________________________________________________  
Layer (type) Output Shape Param #
=================================================================
mask (TimeDist (None, 22, 16, 34, 4) 0
_________________________________________________________________
conv1 (TimeDistributed) (None, 22, 100, 30, 3) 16100
_________________________________________________________________
bn1 (TimeDistributed) (None, 22, 100, 30, 3) 12
_________________________________________________________________
drop1 (Dropout) (None, 22, 100, 30, 3) 0
_________________________________________________________________
conv2 (TimeDistributed) (None, 22, 100, 26, 2) 100100
_________________________________________________________________
bn2 (TimeDistributed) (None, 22, 100, 26, 2) 8
_________________________________________________________________
drop2 (Dropout) (None, 22, 100, 26, 2) 0
_________________________________________________________________
conv3 (TimeDistributed) (None, 22, 100, 22, 1) 100100
_________________________________________________________________
bn3 (TimeDistributed) (None, 22, 100, 22, 1) 4
_________________________________________________________________
drop3 (Dropout) (None, 22, 100, 22, 1) 0
_________________________________________________________________
flatten (TimeDistributed) (None, 22, 2200) 0
_________________________________________________________________
gru (GRU) (None, 22, 256) 1886976
_________________________________________________________________
drop_gru (Dropout) (None, 22, 256) 0
_________________________________________________________________
softmax (Dense) (None, 22, 35) 8995
=================================================================
Total params: 2,112,295
Trainable params: 2,112,283
Non-trainable params: 12
_________________________________________________________________


For mask_value, I tried with either 0 or this mask structure, but neither works and it still trains through all the data with half 0 paddings in it.

Can anyone help me?



B.T.W., I used TimeDistributed here to connect RNN, and I know another one called ConvLSTM2D. Does anyone know the difference? ConvLSTM2D takes much more params for the model, and get training much slower than TimeDistributed...










share|improve this question





























    0















    I have a problem of applying masking layer to CNNs in RNN/LSTM model.



    My data is not original image, but I converted into a shape of (16, 34, 4)(channels_first). The data is sequential, and the longest step length is 22. So for invariant way, I set the timestep as 22. Since it may be shorter than 22 steps, I fill others with np.zeros. However, for 0 padding data, it's about half among all dataset, so with 0 paddings, the training cannot reach a very good result with so much useless data. Then I want to add a mask to cancel these 0 padding data.



    Here is my code.



    mask = np.zeros((16,34,4), dtype = np.int8)  
    input_shape = (22, 16, 34, 4)
    model = Sequential()
    model.add(TimeDistributed(Masking(mask_value=mask), input_shape=input_shape, name = 'mask'))
    model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name = 'conv1'))
    model.add(TimeDistributed(BatchNormalization(), name = 'bn1'))
    model.add(Dropout(0.5, name = 'drop1'))
    model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv2'))
    model.add(TimeDistributed(BatchNormalization(), name = 'bn2'))
    model.add(Dropout(0.5, name = 'drop2'))
    model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv3'))
    model.add(TimeDistributed(BatchNormalization(), name = 'bn3'))
    model.add(Dropout(0.5, name = 'drop3'))
    model.add(TimeDistributed(Flatten(), name = 'flatten'))
    model.add(GRU(256, activation='tanh', return_sequences=True, name = 'gru'))
    model.add(Dropout(0.4, name = 'drop_gru'))
    model.add(Dense(35, activation = 'softmax', name = 'softmax'))
    model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['acc'])


    Here's the model structure.

    model.summary():



    _________________________________________________________________  
    Layer (type) Output Shape Param #
    =================================================================
    mask (TimeDist (None, 22, 16, 34, 4) 0
    _________________________________________________________________
    conv1 (TimeDistributed) (None, 22, 100, 30, 3) 16100
    _________________________________________________________________
    bn1 (TimeDistributed) (None, 22, 100, 30, 3) 12
    _________________________________________________________________
    drop1 (Dropout) (None, 22, 100, 30, 3) 0
    _________________________________________________________________
    conv2 (TimeDistributed) (None, 22, 100, 26, 2) 100100
    _________________________________________________________________
    bn2 (TimeDistributed) (None, 22, 100, 26, 2) 8
    _________________________________________________________________
    drop2 (Dropout) (None, 22, 100, 26, 2) 0
    _________________________________________________________________
    conv3 (TimeDistributed) (None, 22, 100, 22, 1) 100100
    _________________________________________________________________
    bn3 (TimeDistributed) (None, 22, 100, 22, 1) 4
    _________________________________________________________________
    drop3 (Dropout) (None, 22, 100, 22, 1) 0
    _________________________________________________________________
    flatten (TimeDistributed) (None, 22, 2200) 0
    _________________________________________________________________
    gru (GRU) (None, 22, 256) 1886976
    _________________________________________________________________
    drop_gru (Dropout) (None, 22, 256) 0
    _________________________________________________________________
    softmax (Dense) (None, 22, 35) 8995
    =================================================================
    Total params: 2,112,295
    Trainable params: 2,112,283
    Non-trainable params: 12
    _________________________________________________________________


    For mask_value, I tried with either 0 or this mask structure, but neither works and it still trains through all the data with half 0 paddings in it.

    Can anyone help me?



    B.T.W., I used TimeDistributed here to connect RNN, and I know another one called ConvLSTM2D. Does anyone know the difference? ConvLSTM2D takes much more params for the model, and get training much slower than TimeDistributed...










    share|improve this question



























      0












      0








      0








      I have a problem of applying masking layer to CNNs in RNN/LSTM model.



      My data is not original image, but I converted into a shape of (16, 34, 4)(channels_first). The data is sequential, and the longest step length is 22. So for invariant way, I set the timestep as 22. Since it may be shorter than 22 steps, I fill others with np.zeros. However, for 0 padding data, it's about half among all dataset, so with 0 paddings, the training cannot reach a very good result with so much useless data. Then I want to add a mask to cancel these 0 padding data.



      Here is my code.



      mask = np.zeros((16,34,4), dtype = np.int8)  
      input_shape = (22, 16, 34, 4)
      model = Sequential()
      model.add(TimeDistributed(Masking(mask_value=mask), input_shape=input_shape, name = 'mask'))
      model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name = 'conv1'))
      model.add(TimeDistributed(BatchNormalization(), name = 'bn1'))
      model.add(Dropout(0.5, name = 'drop1'))
      model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv2'))
      model.add(TimeDistributed(BatchNormalization(), name = 'bn2'))
      model.add(Dropout(0.5, name = 'drop2'))
      model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv3'))
      model.add(TimeDistributed(BatchNormalization(), name = 'bn3'))
      model.add(Dropout(0.5, name = 'drop3'))
      model.add(TimeDistributed(Flatten(), name = 'flatten'))
      model.add(GRU(256, activation='tanh', return_sequences=True, name = 'gru'))
      model.add(Dropout(0.4, name = 'drop_gru'))
      model.add(Dense(35, activation = 'softmax', name = 'softmax'))
      model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['acc'])


      Here's the model structure.

      model.summary():



      _________________________________________________________________  
      Layer (type) Output Shape Param #
      =================================================================
      mask (TimeDist (None, 22, 16, 34, 4) 0
      _________________________________________________________________
      conv1 (TimeDistributed) (None, 22, 100, 30, 3) 16100
      _________________________________________________________________
      bn1 (TimeDistributed) (None, 22, 100, 30, 3) 12
      _________________________________________________________________
      drop1 (Dropout) (None, 22, 100, 30, 3) 0
      _________________________________________________________________
      conv2 (TimeDistributed) (None, 22, 100, 26, 2) 100100
      _________________________________________________________________
      bn2 (TimeDistributed) (None, 22, 100, 26, 2) 8
      _________________________________________________________________
      drop2 (Dropout) (None, 22, 100, 26, 2) 0
      _________________________________________________________________
      conv3 (TimeDistributed) (None, 22, 100, 22, 1) 100100
      _________________________________________________________________
      bn3 (TimeDistributed) (None, 22, 100, 22, 1) 4
      _________________________________________________________________
      drop3 (Dropout) (None, 22, 100, 22, 1) 0
      _________________________________________________________________
      flatten (TimeDistributed) (None, 22, 2200) 0
      _________________________________________________________________
      gru (GRU) (None, 22, 256) 1886976
      _________________________________________________________________
      drop_gru (Dropout) (None, 22, 256) 0
      _________________________________________________________________
      softmax (Dense) (None, 22, 35) 8995
      =================================================================
      Total params: 2,112,295
      Trainable params: 2,112,283
      Non-trainable params: 12
      _________________________________________________________________


      For mask_value, I tried with either 0 or this mask structure, but neither works and it still trains through all the data with half 0 paddings in it.

      Can anyone help me?



      B.T.W., I used TimeDistributed here to connect RNN, and I know another one called ConvLSTM2D. Does anyone know the difference? ConvLSTM2D takes much more params for the model, and get training much slower than TimeDistributed...










      share|improve this question
















      I have a problem of applying masking layer to CNNs in RNN/LSTM model.



      My data is not original image, but I converted into a shape of (16, 34, 4)(channels_first). The data is sequential, and the longest step length is 22. So for invariant way, I set the timestep as 22. Since it may be shorter than 22 steps, I fill others with np.zeros. However, for 0 padding data, it's about half among all dataset, so with 0 paddings, the training cannot reach a very good result with so much useless data. Then I want to add a mask to cancel these 0 padding data.



      Here is my code.



      mask = np.zeros((16,34,4), dtype = np.int8)  
      input_shape = (22, 16, 34, 4)
      model = Sequential()
      model.add(TimeDistributed(Masking(mask_value=mask), input_shape=input_shape, name = 'mask'))
      model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name = 'conv1'))
      model.add(TimeDistributed(BatchNormalization(), name = 'bn1'))
      model.add(Dropout(0.5, name = 'drop1'))
      model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv2'))
      model.add(TimeDistributed(BatchNormalization(), name = 'bn2'))
      model.add(Dropout(0.5, name = 'drop2'))
      model.add(TimeDistributed(Conv2D(100, (5, 2), data_format = 'channels_first', activation = relu), name ='conv3'))
      model.add(TimeDistributed(BatchNormalization(), name = 'bn3'))
      model.add(Dropout(0.5, name = 'drop3'))
      model.add(TimeDistributed(Flatten(), name = 'flatten'))
      model.add(GRU(256, activation='tanh', return_sequences=True, name = 'gru'))
      model.add(Dropout(0.4, name = 'drop_gru'))
      model.add(Dense(35, activation = 'softmax', name = 'softmax'))
      model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=['acc'])


      Here's the model structure.

      model.summary():



      _________________________________________________________________  
      Layer (type) Output Shape Param #
      =================================================================
      mask (TimeDist (None, 22, 16, 34, 4) 0
      _________________________________________________________________
      conv1 (TimeDistributed) (None, 22, 100, 30, 3) 16100
      _________________________________________________________________
      bn1 (TimeDistributed) (None, 22, 100, 30, 3) 12
      _________________________________________________________________
      drop1 (Dropout) (None, 22, 100, 30, 3) 0
      _________________________________________________________________
      conv2 (TimeDistributed) (None, 22, 100, 26, 2) 100100
      _________________________________________________________________
      bn2 (TimeDistributed) (None, 22, 100, 26, 2) 8
      _________________________________________________________________
      drop2 (Dropout) (None, 22, 100, 26, 2) 0
      _________________________________________________________________
      conv3 (TimeDistributed) (None, 22, 100, 22, 1) 100100
      _________________________________________________________________
      bn3 (TimeDistributed) (None, 22, 100, 22, 1) 4
      _________________________________________________________________
      drop3 (Dropout) (None, 22, 100, 22, 1) 0
      _________________________________________________________________
      flatten (TimeDistributed) (None, 22, 2200) 0
      _________________________________________________________________
      gru (GRU) (None, 22, 256) 1886976
      _________________________________________________________________
      drop_gru (Dropout) (None, 22, 256) 0
      _________________________________________________________________
      softmax (Dense) (None, 22, 35) 8995
      =================================================================
      Total params: 2,112,295
      Trainable params: 2,112,283
      Non-trainable params: 12
      _________________________________________________________________


      For mask_value, I tried with either 0 or this mask structure, but neither works and it still trains through all the data with half 0 paddings in it.

      Can anyone help me?



      B.T.W., I used TimeDistributed here to connect RNN, and I know another one called ConvLSTM2D. Does anyone know the difference? ConvLSTM2D takes much more params for the model, and get training much slower than TimeDistributed...







      keras conv-neural-network lstm mask masking






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Dec 30 '18 at 12:39







      Gao

















      asked Dec 30 '18 at 12:33









      GaoGao

      12




      12
























          0






          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53977616%2fhow-to-apply-masking-layer-to-sequential-cnn-model-in-keras%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53977616%2fhow-to-apply-masking-layer-to-sequential-cnn-model-in-keras%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          2wCbocoj61gb9brKYPgcBZlcAuxA8CZV8jTHrqCrwk,RgZ2U,GVnp9c SJ a7J0N1Yq7,5dAZJ48dLu4oXWSle8o
          h5BauBCFbFOCPo,zFtMBqMWOuX,WRF6Dq H3m Eq7dLbZ2I,SfB1KxW1a7EKr uy ky

          Popular posts from this blog

          Monofisismo

          Angular Downloading a file using contenturl with Basic Authentication

          Olmecas