How to structure video input for LSTM with keras?
I have each frame from multiple videos and their labels in batches - with the use of data generator.
I want to get each frame's CNN features (say VGG) and use them as input to an LSTM (pretty classic).
I understand the concept of the sliding window and step size, but what I don't understand is how do we structure the video in a way that it is aware when a video ends and another begins? In the few examples I could find this is never mentioned and the sliding window is run sequentially on a batch, meaning that it will have windows of one video ending and another beginning as if they are consequent. Putting each video in a batch and padding them is not feasible because 1. The video sizes differ greatly 2. Most videos are longer than batch size.
Another thing I can not get my head around is, all examples focus on predicting the next frame but isn't there a way to do classification directly? Predicting the next frame after 5 frames is a different problem than classifying a sequence of frames, I would think time smoothing or averaging could be an option but is there a better way?).
Thanks.
video keras lstm recurrent-neural-network vgg-net
asked Jan 3 at 16:04
dusadusa
274215
add a comment |
I have each frame from multiple videos and their labels in batches - with the use of data generator.
I want to get each frame's CNN features (say VGG) and use them as input to an LSTM (pretty classic).
I understand the concept of the sliding window and step size, but what I don't understand is how do we structure the video in a way that it is aware when a video ends and another begins? In the few examples I could find this is never mentioned and the sliding window is run sequentially on a batch, meaning that it will have windows of one video ending and another beginning as if they are consequent. Putting each video in a batch and padding them is not feasible because 1. The video sizes differ greatly 2. Most videos are longer than batch size.
Another thing I can not get my head around is, all examples focus on predicting the next frame but isn't there a way to do classification directly? Predicting the next frame after 5 frames is a different problem than classifying a sequence of frames, I would think time smoothing or averaging could be an option but is there a better way?).
Thanks.
video keras lstm recurrent-neural-network vgg-net
asked Jan 3 at 16:04
dusadusa
274215
add a comment |
I have each frame from multiple videos and their labels in batches - with the use of data generator.
I want to get each frame's CNN features (say VGG) and use them as input to an LSTM (pretty classic).
I understand the concept of the sliding window and step size, but what I don't understand is how do we structure the video in a way that it is aware when a video ends and another begins? In the few examples I could find this is never mentioned and the sliding window is run sequentially on a batch, meaning that it will have windows of one video ending and another beginning as if they are consequent. Putting each video in a batch and padding them is not feasible because 1. The video sizes differ greatly 2. Most videos are longer than batch size.
Another thing I can not get my head around is, all examples focus on predicting the next frame but isn't there a way to do classification directly? Predicting the next frame after 5 frames is a different problem than classifying a sequence of frames, I would think time smoothing or averaging could be an option but is there a better way?).
Thanks.
video keras lstm recurrent-neural-network vgg-net
asked Jan 3 at 16:04
dusadusa
274215
I have each frame from multiple videos and their labels in batches - with the use of data generator.
I want to get each frame's CNN features (say VGG) and use them as input to an LSTM (pretty classic).
I understand the concept of the sliding window and step size, but what I don't understand is how do we structure the video in a way that it is aware when a video ends and another begins? In the few examples I could find this is never mentioned and the sliding window is run sequentially on a batch, meaning that it will have windows of one video ending and another beginning as if they are consequent. Putting each video in a batch and padding them is not feasible because 1. The video sizes differ greatly 2. Most videos are longer than batch size.
Another thing I can not get my head around is, all examples focus on predicting the next frame but isn't there a way to do classification directly? Predicting the next frame after 5 frames is a different problem than classifying a sequence of frames, I would think time smoothing or averaging could be an option but is there a better way?).
Thanks.
video keras lstm recurrent-neural-network vgg-net
video keras lstm recurrent-neural-network vgg-net
asked Jan 3 at 16:04
dusadusa
274215
asked Jan 3 at 16:04
dusadusa
274215
asked Jan 3 at 16:04
dusadusa
274215
asked Jan 3 at 16:04
dusadusa
274215
asked Jan 3 at 16:04
dusadusa
274215
274215
add a comment |
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54025863%2fhow-to-structure-video-input-for-lstm-with-keras%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54025863%2fhow-to-structure-video-input-for-lstm-with-keras%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
