Custom Keras loss function that conditionally creates a zero gradient
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
0
My problem is I don't want the weights to be adjusted if y_true takes certain values. I do not want to simply remove those examples from training data because of the nature of the RNN I am trying to use.
Is there a way to write a conditional loss function in Keras with this behavior?
For example: if y_true is negative then apply zero gradient so that parameters in the model do not change, if y_true is positive loss = losses.mean_squared_error(y_true, y_pred).
python tensorflow machine-learning keras loss-function
edited Jan 4 at 13:48
today
11.7k22541
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
add a comment |
0
My problem is I don't want the weights to be adjusted if y_true takes certain values. I do not want to simply remove those examples from training data because of the nature of the RNN I am trying to use.
Is there a way to write a conditional loss function in Keras with this behavior?
For example: if y_true is negative then apply zero gradient so that parameters in the model do not change, if y_true is positive loss = losses.mean_squared_error(y_true, y_pred).
python tensorflow machine-learning keras loss-function
edited Jan 4 at 13:48
today
11.7k22541
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
Include another input variablemaskwhich is zero for the specific values ofy_true?
– Parag S. Chandakkar
Jan 4 at 0:41
add a comment |
0
0
0
My problem is I don't want the weights to be adjusted if y_true takes certain values. I do not want to simply remove those examples from training data because of the nature of the RNN I am trying to use.
Is there a way to write a conditional loss function in Keras with this behavior?
For example: if y_true is negative then apply zero gradient so that parameters in the model do not change, if y_true is positive loss = losses.mean_squared_error(y_true, y_pred).
python tensorflow machine-learning keras loss-function
edited Jan 4 at 13:48
today
11.7k22541
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
My problem is I don't want the weights to be adjusted if y_true takes certain values. I do not want to simply remove those examples from training data because of the nature of the RNN I am trying to use.
Is there a way to write a conditional loss function in Keras with this behavior?
For example: if y_true is negative then apply zero gradient so that parameters in the model do not change, if y_true is positive loss = losses.mean_squared_error(y_true, y_pred).
python tensorflow machine-learning keras loss-function
python tensorflow machine-learning keras loss-function
edited Jan 4 at 13:48
today
11.7k22541
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
edited Jan 4 at 13:48
today
11.7k22541
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
edited Jan 4 at 13:48
today
11.7k22541
edited Jan 4 at 13:48
today
11.7k22541
edited Jan 4 at 13:48
today
11.7k22541
11.7k22541
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
asked Jan 4 at 0:25
Nick MerrillNick Merrill
31
31
Include another input variablemaskwhich is zero for the specific values ofy_true?
– Parag S. Chandakkar
Jan 4 at 0:41
add a comment |
Include another input variablemaskwhich is zero for the specific values ofy_true?
– Parag S. Chandakkar
Jan 4 at 0:41
Include another input variable
mask which is zero for the specific values of y_true?– Parag S. Chandakkar
Jan 4 at 0:41
Include another input variable
mask which is zero for the specific values of y_true?– Parag S. Chandakkar
Jan 4 at 0:41
add a comment |
2 Answers
2
active
oldest
votes
0
You can define a custom loss function and simply use K.switch to conditionally get zero loss:
from keras import backend as K
from keras import losses
def custom_loss(y_true, y_pred):
loss = losses.mean_squared_error(y_true, y_pred)
return K.switch(K.flatten(K.equal(y_true, 0.)), K.zeros_like(loss), loss)
Test:
from keras import models
from keras import layers
model = models.Sequential()
model.add(layers.Dense(1, input_shape=(1,)))
model.compile(loss=custom_loss, optimizer='adam')
weights, bias = model.layers[0].get_weights()
x = np.array([1, 2, 3])
y = np.array([0, 0, 0])
model.train_on_batch(x, y)
# check if the parameters has not changed after training on the batch
>>> (weights == model.layers[0].get_weights()[0]).all()
True
>>> (bias == model.layers[0].get_weights()[1]).all()
True
answered Jan 4 at 10:53
todaytoday
11.7k22541
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill I have updated my answer and added a test as well as some minor updates to thecustom_lossfunction.
– today
Jan 4 at 13:43
|
show 1 more comment
0
Since the y's are in batches, you need to select those from the batch which are non-zero in the custom loss function
def myloss(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
return losses.mean_squared_error(y_true, y_pred)
Then it can be used as such:
model = keras.Sequential([Dense(32, input_shape=(2,)), Dense(1)])
model.compile('adam', loss=myloss)
x = np.random.randn(2, 2)
y = np.array([1, 0])
model.fit(x, y)
But you might need extra logic in the loss function in case all y_true in the batch were zero, in this case, the loss function can be modified as such:
def myloss2(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
loss = tf.cond(tf.equal(tf.shape(y_pred)[0], 0), lambda: tf.constant(0, dtype=tf.float32), lambda: losses.mean_squared_error(y_true, y_pred))
return loss
answered Jan 4 at 1:04
GergesGerges
3,0931821
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54031644%2fcustom-keras-loss-function-that-conditionally-creates-a-zero-gradient%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
You can define a custom loss function and simply use K.switch to conditionally get zero loss:
from keras import backend as K
from keras import losses
def custom_loss(y_true, y_pred):
loss = losses.mean_squared_error(y_true, y_pred)
return K.switch(K.flatten(K.equal(y_true, 0.)), K.zeros_like(loss), loss)
Test:
from keras import models
from keras import layers
model = models.Sequential()
model.add(layers.Dense(1, input_shape=(1,)))
model.compile(loss=custom_loss, optimizer='adam')
weights, bias = model.layers[0].get_weights()
x = np.array([1, 2, 3])
y = np.array([0, 0, 0])
model.train_on_batch(x, y)
# check if the parameters has not changed after training on the batch
>>> (weights == model.layers[0].get_weights()[0]).all()
True
>>> (bias == model.layers[0].get_weights()[1]).all()
True
answered Jan 4 at 10:53
todaytoday
11.7k22541
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill I have updated my answer and added a test as well as some minor updates to thecustom_lossfunction.
– today
Jan 4 at 13:43
|
show 1 more comment
0
You can define a custom loss function and simply use K.switch to conditionally get zero loss:
from keras import backend as K
from keras import losses
def custom_loss(y_true, y_pred):
loss = losses.mean_squared_error(y_true, y_pred)
return K.switch(K.flatten(K.equal(y_true, 0.)), K.zeros_like(loss), loss)
Test:
from keras import models
from keras import layers
model = models.Sequential()
model.add(layers.Dense(1, input_shape=(1,)))
model.compile(loss=custom_loss, optimizer='adam')
weights, bias = model.layers[0].get_weights()
x = np.array([1, 2, 3])
y = np.array([0, 0, 0])
model.train_on_batch(x, y)
# check if the parameters has not changed after training on the batch
>>> (weights == model.layers[0].get_weights()[0]).all()
True
>>> (bias == model.layers[0].get_weights()[1]).all()
True
answered Jan 4 at 10:53
todaytoday
11.7k22541
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill I have updated my answer and added a test as well as some minor updates to thecustom_lossfunction.
– today
Jan 4 at 13:43
|
show 1 more comment
0
0
0
You can define a custom loss function and simply use K.switch to conditionally get zero loss:
from keras import backend as K
from keras import losses
def custom_loss(y_true, y_pred):
loss = losses.mean_squared_error(y_true, y_pred)
return K.switch(K.flatten(K.equal(y_true, 0.)), K.zeros_like(loss), loss)
Test:
from keras import models
from keras import layers
model = models.Sequential()
model.add(layers.Dense(1, input_shape=(1,)))
model.compile(loss=custom_loss, optimizer='adam')
weights, bias = model.layers[0].get_weights()
x = np.array([1, 2, 3])
y = np.array([0, 0, 0])
model.train_on_batch(x, y)
# check if the parameters has not changed after training on the batch
>>> (weights == model.layers[0].get_weights()[0]).all()
True
>>> (bias == model.layers[0].get_weights()[1]).all()
True
answered Jan 4 at 10:53
todaytoday
11.7k22541
You can define a custom loss function and simply use K.switch to conditionally get zero loss:
from keras import backend as K
from keras import losses
def custom_loss(y_true, y_pred):
loss = losses.mean_squared_error(y_true, y_pred)
return K.switch(K.flatten(K.equal(y_true, 0.)), K.zeros_like(loss), loss)
Test:
from keras import models
from keras import layers
model = models.Sequential()
model.add(layers.Dense(1, input_shape=(1,)))
model.compile(loss=custom_loss, optimizer='adam')
weights, bias = model.layers[0].get_weights()
x = np.array([1, 2, 3])
y = np.array([0, 0, 0])
model.train_on_batch(x, y)
# check if the parameters has not changed after training on the batch
>>> (weights == model.layers[0].get_weights()[0]).all()
True
>>> (bias == model.layers[0].get_weights()[1]).all()
True
answered Jan 4 at 10:53
todaytoday
11.7k22541
edited Jan 4 at 14:05
answered Jan 4 at 10:53
todaytoday
11.7k22541
answered Jan 4 at 10:53
todaytoday
11.7k22541
answered Jan 4 at 10:53
todaytoday
11.7k22541
11.7k22541
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill I have updated my answer and added a test as well as some minor updates to thecustom_lossfunction.
– today
Jan 4 at 13:43
|
show 1 more comment
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill I have updated my answer and added a test as well as some minor updates to thecustom_lossfunction.
– today
Jan 4 at 13:43
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
I'm not sure how Keras calculates gradients, does zero loss automatically mean zero gradient?
– Nick Merrill
Jan 4 at 13:24
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
@NickMerrill Well, the gradient of a constant, including zero, with respect to any variable is zero, right?
– today
Jan 4 at 13:27
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
Of course, it's just kind of a black box to me how keras performs this calculation. Is there a simply way to "see" the gradient or to check and weight or bias to see if it is changing?
– Nick Merrill
Jan 4 at 13:28
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill Sure, wait a few minutes. I am writing an update.
– today
Jan 4 at 13:29
@NickMerrill I have updated my answer and added a test as well as some minor updates to the
custom_loss function.– today
Jan 4 at 13:43
@NickMerrill I have updated my answer and added a test as well as some minor updates to the
custom_loss function.– today
Jan 4 at 13:43
|
show 1 more comment
0
Since the y's are in batches, you need to select those from the batch which are non-zero in the custom loss function
def myloss(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
return losses.mean_squared_error(y_true, y_pred)
Then it can be used as such:
model = keras.Sequential([Dense(32, input_shape=(2,)), Dense(1)])
model.compile('adam', loss=myloss)
x = np.random.randn(2, 2)
y = np.array([1, 0])
model.fit(x, y)
But you might need extra logic in the loss function in case all y_true in the batch were zero, in this case, the loss function can be modified as such:
def myloss2(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
loss = tf.cond(tf.equal(tf.shape(y_pred)[0], 0), lambda: tf.constant(0, dtype=tf.float32), lambda: losses.mean_squared_error(y_true, y_pred))
return loss
answered Jan 4 at 1:04
GergesGerges
3,0931821
add a comment |
0
Since the y's are in batches, you need to select those from the batch which are non-zero in the custom loss function
def myloss(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
return losses.mean_squared_error(y_true, y_pred)
Then it can be used as such:
model = keras.Sequential([Dense(32, input_shape=(2,)), Dense(1)])
model.compile('adam', loss=myloss)
x = np.random.randn(2, 2)
y = np.array([1, 0])
model.fit(x, y)
But you might need extra logic in the loss function in case all y_true in the batch were zero, in this case, the loss function can be modified as such:
def myloss2(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
loss = tf.cond(tf.equal(tf.shape(y_pred)[0], 0), lambda: tf.constant(0, dtype=tf.float32), lambda: losses.mean_squared_error(y_true, y_pred))
return loss
answered Jan 4 at 1:04
GergesGerges
3,0931821
add a comment |
0
0
0
Since the y's are in batches, you need to select those from the batch which are non-zero in the custom loss function
def myloss(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
return losses.mean_squared_error(y_true, y_pred)
Then it can be used as such:
model = keras.Sequential([Dense(32, input_shape=(2,)), Dense(1)])
model.compile('adam', loss=myloss)
x = np.random.randn(2, 2)
y = np.array([1, 0])
model.fit(x, y)
But you might need extra logic in the loss function in case all y_true in the batch were zero, in this case, the loss function can be modified as such:
def myloss2(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
loss = tf.cond(tf.equal(tf.shape(y_pred)[0], 0), lambda: tf.constant(0, dtype=tf.float32), lambda: losses.mean_squared_error(y_true, y_pred))
return loss
answered Jan 4 at 1:04
GergesGerges
3,0931821
Since the y's are in batches, you need to select those from the batch which are non-zero in the custom loss function
def myloss(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
return losses.mean_squared_error(y_true, y_pred)
Then it can be used as such:
model = keras.Sequential([Dense(32, input_shape=(2,)), Dense(1)])
model.compile('adam', loss=myloss)
x = np.random.randn(2, 2)
y = np.array([1, 0])
model.fit(x, y)
But you might need extra logic in the loss function in case all y_true in the batch were zero, in this case, the loss function can be modified as such:
def myloss2(y_true, y_pred):
idx = tf.not_equal(y_true, 0)
y_true = tf.boolean_mask(y_true, idx)
y_pred = tf.boolean_mask(y_pred, idx)
loss = tf.cond(tf.equal(tf.shape(y_pred)[0], 0), lambda: tf.constant(0, dtype=tf.float32), lambda: losses.mean_squared_error(y_true, y_pred))
return loss
answered Jan 4 at 1:04
GergesGerges
3,0931821
edited Jan 4 at 1:10
answered Jan 4 at 1:04
GergesGerges
3,0931821
answered Jan 4 at 1:04
GergesGerges
3,0931821
answered Jan 4 at 1:04
GergesGerges
3,0931821
3,0931821
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54031644%2fcustom-keras-loss-function-that-conditionally-creates-a-zero-gradient%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Include another input variable
maskwhich is zero for the specific values ofy_true?– Parag S. Chandakkar
Jan 4 at 0:41