Redis - Which datatype would be most efficient way to store data

Multi tool use
I am not exactly sure which datatype/approach would be best for given the following scenarios.
I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).
public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}
I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.
I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)
number 2 above Can also be done with paging but filtered by one of the fields first then return results
I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.
So in short, I like my list of objects stored to be retrieved by a
Retrieve a single item by an ID (property of the json object
Paged lists without filters
Paged lists with paging and filtering of json objects
At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)
I am writing the code in .NET so if that makes any difference.
c# .net caching redis
|
show 5 more comments
I am not exactly sure which datatype/approach would be best for given the following scenarios.
I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).
public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}
I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.
I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)
number 2 above Can also be done with paging but filtered by one of the fields first then return results
I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.
So in short, I like my list of objects stored to be retrieved by a
Retrieve a single item by an ID (property of the json object
Paged lists without filters
Paged lists with paging and filtering of json objects
At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)
I am writing the code in .NET so if that makes any difference.
c# .net caching redis
Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.
– Nate
Dec 30 '18 at 20:36
@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds
– Zoinky
Dec 30 '18 at 20:41
If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.
– Nate
Dec 30 '18 at 20:49
1
Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.
– Nate
Dec 30 '18 at 20:50
@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.
– Zoinky
Dec 30 '18 at 20:57
|
show 5 more comments
I am not exactly sure which datatype/approach would be best for given the following scenarios.
I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).
public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}
I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.
I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)
number 2 above Can also be done with paging but filtered by one of the fields first then return results
I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.
So in short, I like my list of objects stored to be retrieved by a
Retrieve a single item by an ID (property of the json object
Paged lists without filters
Paged lists with paging and filtering of json objects
At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)
I am writing the code in .NET so if that makes any difference.
c# .net caching redis
I am not exactly sure which datatype/approach would be best for given the following scenarios.
I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).
public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}
I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.
I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)
number 2 above Can also be done with paging but filtered by one of the fields first then return results
I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.
So in short, I like my list of objects stored to be retrieved by a
Retrieve a single item by an ID (property of the json object
Paged lists without filters
Paged lists with paging and filtering of json objects
At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)
I am writing the code in .NET so if that makes any difference.
c# .net caching redis
c# .net caching redis
asked Dec 30 '18 at 20:26
ZoinkyZoinky
1,31132037
1,31132037
Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.
– Nate
Dec 30 '18 at 20:36
@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds
– Zoinky
Dec 30 '18 at 20:41
If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.
– Nate
Dec 30 '18 at 20:49
1
Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.
– Nate
Dec 30 '18 at 20:50
@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.
– Zoinky
Dec 30 '18 at 20:57
|
show 5 more comments
Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.
– Nate
Dec 30 '18 at 20:36
@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds
– Zoinky
Dec 30 '18 at 20:41
If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.
– Nate
Dec 30 '18 at 20:49
1
Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.
– Nate
Dec 30 '18 at 20:50
@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.
– Zoinky
Dec 30 '18 at 20:57
Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.
– Nate
Dec 30 '18 at 20:36
Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.
– Nate
Dec 30 '18 at 20:36
@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds
– Zoinky
Dec 30 '18 at 20:41
@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds
– Zoinky
Dec 30 '18 at 20:41
If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.
– Nate
Dec 30 '18 at 20:49
If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.
– Nate
Dec 30 '18 at 20:49
1
1
Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.
– Nate
Dec 30 '18 at 20:50
Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.
– Nate
Dec 30 '18 at 20:50
@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.
– Zoinky
Dec 30 '18 at 20:57
@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.
– Zoinky
Dec 30 '18 at 20:57
|
show 5 more comments
1 Answer
1
active
oldest
votes
Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.
If I understand your requirements correctly, here is how I'd approach:
Store objects in a sorted set (if you the returned objects in a specific order)
You can now querying single and unfiltered objects by native Redis commands:
Redis - Sorted set, find item by property valueFor filtered objects you can look into Lua scripts
Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis
You can also check ignite, it has a couple of features built in that may be of interest to you:
Binary marshaller :
"It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
https://apacheignite.readme.io/docs/binary-marshallerRead/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53981162%2fredis-which-datatype-would-be-most-efficient-way-to-store-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.
If I understand your requirements correctly, here is how I'd approach:
Store objects in a sorted set (if you the returned objects in a specific order)
You can now querying single and unfiltered objects by native Redis commands:
Redis - Sorted set, find item by property valueFor filtered objects you can look into Lua scripts
Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis
You can also check ignite, it has a couple of features built in that may be of interest to you:
Binary marshaller :
"It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
https://apacheignite.readme.io/docs/binary-marshallerRead/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html
add a comment |
Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.
If I understand your requirements correctly, here is how I'd approach:
Store objects in a sorted set (if you the returned objects in a specific order)
You can now querying single and unfiltered objects by native Redis commands:
Redis - Sorted set, find item by property valueFor filtered objects you can look into Lua scripts
Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis
You can also check ignite, it has a couple of features built in that may be of interest to you:
Binary marshaller :
"It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
https://apacheignite.readme.io/docs/binary-marshallerRead/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html
add a comment |
Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.
If I understand your requirements correctly, here is how I'd approach:
Store objects in a sorted set (if you the returned objects in a specific order)
You can now querying single and unfiltered objects by native Redis commands:
Redis - Sorted set, find item by property valueFor filtered objects you can look into Lua scripts
Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis
You can also check ignite, it has a couple of features built in that may be of interest to you:
Binary marshaller :
"It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
https://apacheignite.readme.io/docs/binary-marshallerRead/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html
Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.
If I understand your requirements correctly, here is how I'd approach:
Store objects in a sorted set (if you the returned objects in a specific order)
You can now querying single and unfiltered objects by native Redis commands:
Redis - Sorted set, find item by property valueFor filtered objects you can look into Lua scripts
Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis
You can also check ignite, it has a couple of features built in that may be of interest to you:
Binary marshaller :
"It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
https://apacheignite.readme.io/docs/binary-marshallerRead/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html
answered Dec 31 '18 at 18:24
rainhackerrainhacker
1038
1038
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53981162%2fredis-which-datatype-would-be-most-efficient-way-to-store-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
7yIkzFwQ6,ZqD Jh6in4xb5,3A,mdcfjO3MqesbdUoG6qgCUmiEXQYt,HgZgzI2V s0oJX84DNhEDKbrSRSOkRmwsTm,wxhYguRxcBR3
Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.
– Nate
Dec 30 '18 at 20:36
@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds
– Zoinky
Dec 30 '18 at 20:41
If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.
– Nate
Dec 30 '18 at 20:49
1
Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.
– Nate
Dec 30 '18 at 20:50
@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.
– Zoinky
Dec 30 '18 at 20:57