Solr relevancy score different on replicated nodes
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
Version Solr 7.4.0 zookeeper 3.4.11
Achitecture Two boxes Machine-1,Machine-2 holding single instances of solr
We are having a collection which was single shard and single replica i.e s=1 and rf=1
Few days back we tried to add replica to it.But the score for same query is coming different from different replicas.
http://Machine-1:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.418847,"docs":[
whereas on another machine(replica)
http://Machine-2:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.4952264,"docs":[
The maxScore is different.
Relevancy gets affected due to sharding but replication was not expected as same documents get copied to other node. score explaination gives issue with docCount and docFreq uneven.
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.050635000
docCount :10020.000000000
docFreq :3504.0000000
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.068795100
docCount :10291.000000000
docFreq :3534.0000000
Update:Tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
How can we correct the original collection.
solr solrcloud
asked Jan 4 at 9:22
AbrocksAbrocks
13
add a comment |
Version Solr 7.4.0 zookeeper 3.4.11
Achitecture Two boxes Machine-1,Machine-2 holding single instances of solr
We are having a collection which was single shard and single replica i.e s=1 and rf=1
Few days back we tried to add replica to it.But the score for same query is coming different from different replicas.
http://Machine-1:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.418847,"docs":[
whereas on another machine(replica)
http://Machine-2:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.4952264,"docs":[
The maxScore is different.
Relevancy gets affected due to sharding but replication was not expected as same documents get copied to other node. score explaination gives issue with docCount and docFreq uneven.
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.050635000
docCount :10020.000000000
docFreq :3504.0000000
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.068795100
docCount :10291.000000000
docFreq :3534.0000000
Update:Tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
How can we correct the original collection.
solr solrcloud
asked Jan 4 at 9:22
AbrocksAbrocks
13
Initial guess - the number of deleted documents differ on the two replicas (they'll still affect the score until they've been expunged). Their mergefactors might be different, or a document failed being committed to one of the replicas while indexing - and depending on your settings when submitting documents - that might not be propagated as a fatal error. Deleting the replica and adding it again should force replication of the complete index again. Issue a fetch all query to all the replicas and see if the number of documents are different (since the docCount differ)
– MatsLindh
Jan 4 at 12:35
machine-1:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-2:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-1:8983/solr/MyTestCollection/select?q=*:*&shards=machine-2:8983/solr/MyTestCollection All giving same number of documents. We can delete replicas but which one should be deleted.How we can know which node is giving correct score.
– Abrocks
Jan 4 at 13:43
Collection state below "shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{ "core_node2":{"core":"MyTestCollection_shard1_replica_n1","base_url":"machine-1.com:8983/…", "type":"NRT","force_set_state":"false","leader":"true"},"core_node4":{"core":"MyTestCollection_shard1_replica_n3","base_url":"machine-2:8983/…", "force_set_state":"false"}}}},
– Abrocks
Jan 4 at 13:45
add a comment |
Version Solr 7.4.0 zookeeper 3.4.11
Achitecture Two boxes Machine-1,Machine-2 holding single instances of solr
We are having a collection which was single shard and single replica i.e s=1 and rf=1
Few days back we tried to add replica to it.But the score for same query is coming different from different replicas.
http://Machine-1:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.418847,"docs":[
whereas on another machine(replica)
http://Machine-2:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.4952264,"docs":[
The maxScore is different.
Relevancy gets affected due to sharding but replication was not expected as same documents get copied to other node. score explaination gives issue with docCount and docFreq uneven.
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.050635000
docCount :10020.000000000
docFreq :3504.0000000
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.068795100
docCount :10291.000000000
docFreq :3534.0000000
Update:Tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
How can we correct the original collection.
solr solrcloud
asked Jan 4 at 9:22
AbrocksAbrocks
13
Version Solr 7.4.0 zookeeper 3.4.11
Achitecture Two boxes Machine-1,Machine-2 holding single instances of solr
We are having a collection which was single shard and single replica i.e s=1 and rf=1
Few days back we tried to add replica to it.But the score for same query is coming different from different replicas.
http://Machine-1:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.418847,"docs":[
whereas on another machine(replica)
http://Machine-2:8983/solr/MyTestCollection/select?q=%22data%22+OR+(data)&rows=10&fl=score&defType=edismax&qf=search_field+content&wt=json
"response":{"numFound":5836,"start":0,"maxScore":4.4952264,"docs":[
The maxScore is different.
Relevancy gets affected due to sharding but replication was not expected as same documents get copied to other node. score explaination gives issue with docCount and docFreq uneven.
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.050635000
docCount :10020.000000000
docFreq :3504.0000000
idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:
1.068795100
docCount :10291.000000000
docFreq :3534.0000000
Update:Tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
How can we correct the original collection.
solr solrcloud
solr solrcloud
asked Jan 4 at 9:22
AbrocksAbrocks
13
asked Jan 4 at 9:22
AbrocksAbrocks
13
edited Jan 4 at 12:29
Abrocks
asked Jan 4 at 9:22
AbrocksAbrocks
13
asked Jan 4 at 9:22
AbrocksAbrocks
13
asked Jan 4 at 9:22
AbrocksAbrocks
13
13
Initial guess - the number of deleted documents differ on the two replicas (they'll still affect the score until they've been expunged). Their mergefactors might be different, or a document failed being committed to one of the replicas while indexing - and depending on your settings when submitting documents - that might not be propagated as a fatal error. Deleting the replica and adding it again should force replication of the complete index again. Issue a fetch all query to all the replicas and see if the number of documents are different (since the docCount differ)
– MatsLindh
Jan 4 at 12:35
machine-1:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-2:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-1:8983/solr/MyTestCollection/select?q=*:*&shards=machine-2:8983/solr/MyTestCollection All giving same number of documents. We can delete replicas but which one should be deleted.How we can know which node is giving correct score.
– Abrocks
Jan 4 at 13:43
Collection state below "shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{ "core_node2":{"core":"MyTestCollection_shard1_replica_n1","base_url":"machine-1.com:8983/…", "type":"NRT","force_set_state":"false","leader":"true"},"core_node4":{"core":"MyTestCollection_shard1_replica_n3","base_url":"machine-2:8983/…", "force_set_state":"false"}}}},
– Abrocks
Jan 4 at 13:45
add a comment |
Initial guess - the number of deleted documents differ on the two replicas (they'll still affect the score until they've been expunged). Their mergefactors might be different, or a document failed being committed to one of the replicas while indexing - and depending on your settings when submitting documents - that might not be propagated as a fatal error. Deleting the replica and adding it again should force replication of the complete index again. Issue a fetch all query to all the replicas and see if the number of documents are different (since the docCount differ)
– MatsLindh
Jan 4 at 12:35
machine-1:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-2:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-1:8983/solr/MyTestCollection/select?q=*:*&shards=machine-2:8983/solr/MyTestCollection All giving same number of documents. We can delete replicas but which one should be deleted.How we can know which node is giving correct score.
– Abrocks
Jan 4 at 13:43
Collection state below "shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{ "core_node2":{"core":"MyTestCollection_shard1_replica_n1","base_url":"machine-1.com:8983/…", "type":"NRT","force_set_state":"false","leader":"true"},"core_node4":{"core":"MyTestCollection_shard1_replica_n3","base_url":"machine-2:8983/…", "force_set_state":"false"}}}},
– Abrocks
Jan 4 at 13:45
Initial guess - the number of deleted documents differ on the two replicas (they'll still affect the score until they've been expunged). Their mergefactors might be different, or a document failed being committed to one of the replicas while indexing - and depending on your settings when submitting documents - that might not be propagated as a fatal error. Deleting the replica and adding it again should force replication of the complete index again. Issue a fetch all query to all the replicas and see if the number of documents are different (since the docCount differ)
– MatsLindh
Jan 4 at 12:35
Initial guess - the number of deleted documents differ on the two replicas (they'll still affect the score until they've been expunged). Their mergefactors might be different, or a document failed being committed to one of the replicas while indexing - and depending on your settings when submitting documents - that might not be propagated as a fatal error. Deleting the replica and adding it again should force replication of the complete index again. Issue a fetch all query to all the replicas and see if the number of documents are different (since the docCount differ)
– MatsLindh
Jan 4 at 12:35
machine-1:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-2:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-1:8983/solr/MyTestCollection/select?q=*:*&shards=machine-2:8983/solr/MyTestCollection All giving same number of documents. We can delete replicas but which one should be deleted.How we can know which node is giving correct score.
– Abrocks
Jan 4 at 13:43
machine-1:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-2:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-1:8983/solr/MyTestCollection/select?q=*:*&shards=machine-2:8983/solr/MyTestCollection All giving same number of documents. We can delete replicas but which one should be deleted.How we can know which node is giving correct score.
– Abrocks
Jan 4 at 13:43
Collection state below "shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{ "core_node2":{"core":"MyTestCollection_shard1_replica_n1","base_url":"machine-1.com:8983/…", "type":"NRT","force_set_state":"false","leader":"true"},"core_node4":{"core":"MyTestCollection_shard1_replica_n3","base_url":"machine-2:8983/…", "force_set_state":"false"}}}},
– Abrocks
Jan 4 at 13:45
Collection state below "shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{ "core_node2":{"core":"MyTestCollection_shard1_replica_n1","base_url":"machine-1.com:8983/…", "type":"NRT","force_set_state":"false","leader":"true"},"core_node4":{"core":"MyTestCollection_shard1_replica_n3","base_url":"machine-2:8983/…", "force_set_state":"false"}}}},
– Abrocks
Jan 4 at 13:45
add a comment |
1 Answer
1
active
oldest
votes
if you are doing this while indexing is going on, it is possible that the master has more docs than the replica (the replica is behind, always catching up), so the doc numbers might vary slightly. If you get this when no indexing is going on, and you have commited, then it should be a bug.
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54036079%2fsolr-relevancy-score-different-on-replicated-nodes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
if you are doing this while indexing is going on, it is possible that the master has more docs than the replica (the replica is behind, always catching up), so the doc numbers might vary slightly. If you get this when no indexing is going on, and you have commited, then it should be a bug.
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
add a comment |
if you are doing this while indexing is going on, it is possible that the master has more docs than the replica (the replica is behind, always catching up), so the doc numbers might vary slightly. If you get this when no indexing is going on, and you have commited, then it should be a bug.
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
add a comment |
if you are doing this while indexing is going on, it is possible that the master has more docs than the replica (the replica is behind, always catching up), so the doc numbers might vary slightly. If you get this when no indexing is going on, and you have commited, then it should be a bug.
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
if you are doing this while indexing is going on, it is possible that the master has more docs than the replica (the replica is behind, always catching up), so the doc numbers might vary slightly. If you get this when no indexing is going on, and you have commited, then it should be a bug.
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
answered Jan 4 at 12:29
PersimmoniumPersimmonium
12.5k103968
12.5k103968
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
add a comment |
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
We didn't do it during indexing.Also we tried the same on different collection.Both instances give same score.Seems like a issue with particular collection.
– Abrocks
Jan 4 at 12:31
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
you sure both replicas have commited all docs? if they have different commit settings they might differ
– Persimmonium
Jan 4 at 14:45
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
Yes, because hitting all query on both replicas give same no of docs..
– Abrocks
Jan 4 at 18:44
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54036079%2fsolr-relevancy-score-different-on-replicated-nodes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Initial guess - the number of deleted documents differ on the two replicas (they'll still affect the score until they've been expunged). Their mergefactors might be different, or a document failed being committed to one of the replicas while indexing - and depending on your settings when submitting documents - that might not be propagated as a fatal error. Deleting the replica and adding it again should force replication of the complete index again. Issue a fetch all query to all the replicas and see if the number of documents are different (since the docCount differ)
– MatsLindh
Jan 4 at 12:35
machine-1:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-2:8983/solr/MyTestCollection/select?q=*:*&rows=0 machine-1:8983/solr/MyTestCollection/select?q=*:*&shards=machine-2:8983/solr/MyTestCollection All giving same number of documents. We can delete replicas but which one should be deleted.How we can know which node is giving correct score.
– Abrocks
Jan 4 at 13:43
Collection state below "shards":{"shard1":{"range":"80000000-7fffffff","state":"active","replicas":{ "core_node2":{"core":"MyTestCollection_shard1_replica_n1","base_url":"machine-1.com:8983/…", "type":"NRT","force_set_state":"false","leader":"true"},"core_node4":{"core":"MyTestCollection_shard1_replica_n3","base_url":"machine-2:8983/…", "force_set_state":"false"}}}},
– Abrocks
Jan 4 at 13:45