Redis - Which datatype would be most efficient way to store data

Multi tool use
Multi tool use












0















I am not exactly sure which datatype/approach would be best for given the following scenarios.



I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).



public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}



  1. I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.


  2. I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)


  3. number 2 above Can also be done with paging but filtered by one of the fields first then return results



I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.



So in short, I like my list of objects stored to be retrieved by a




  1. Retrieve a single item by an ID (property of the json object


  2. Paged lists without filters


  3. Paged lists with paging and filtering of json objects


  4. At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)



I am writing the code in .NET so if that makes any difference.










share|improve this question























  • Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.

    – Nate
    Dec 30 '18 at 20:36











  • @nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds

    – Zoinky
    Dec 30 '18 at 20:41











  • If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.

    – Nate
    Dec 30 '18 at 20:49






  • 1





    Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.

    – Nate
    Dec 30 '18 at 20:50











  • @Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.

    – Zoinky
    Dec 30 '18 at 20:57


















0















I am not exactly sure which datatype/approach would be best for given the following scenarios.



I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).



public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}



  1. I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.


  2. I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)


  3. number 2 above Can also be done with paging but filtered by one of the fields first then return results



I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.



So in short, I like my list of objects stored to be retrieved by a




  1. Retrieve a single item by an ID (property of the json object


  2. Paged lists without filters


  3. Paged lists with paging and filtering of json objects


  4. At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)



I am writing the code in .NET so if that makes any difference.










share|improve this question























  • Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.

    – Nate
    Dec 30 '18 at 20:36











  • @nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds

    – Zoinky
    Dec 30 '18 at 20:41











  • If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.

    – Nate
    Dec 30 '18 at 20:49






  • 1





    Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.

    – Nate
    Dec 30 '18 at 20:50











  • @Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.

    – Zoinky
    Dec 30 '18 at 20:57
















0












0








0








I am not exactly sure which datatype/approach would be best for given the following scenarios.



I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).



public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}



  1. I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.


  2. I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)


  3. number 2 above Can also be done with paging but filtered by one of the fields first then return results



I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.



So in short, I like my list of objects stored to be retrieved by a




  1. Retrieve a single item by an ID (property of the json object


  2. Paged lists without filters


  3. Paged lists with paging and filtering of json objects


  4. At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)



I am writing the code in .NET so if that makes any difference.










share|improve this question














I am not exactly sure which datatype/approach would be best for given the following scenarios.



I will be storing full json objects and each of these objects will have multiple properties, one of them will be an ID (int field).



public class Event
{
public int EventId {get; set}
public DateTime EventDate {get; set;}
public string Title {get; set;}
public int TypeId {get; set;}
}



  1. I will need to be able to look up individual objects by this id, I assume this will be just be stored as a key/value pair, key being "something"+id and the value being a serialized json object.


  2. I would like to be able to get list of the above objects in a paged manner, say first page, and page size is 20. (Hashset or Sorted Set)


  3. number 2 above Can also be done with paging but filtered by one of the fields first then return results



I would like to only have one copy of each json object to satisfy the above scenarios, from everything I read so far it seems that I will be creating multiple copies of each object to satisfy all of the above scenarios.



So in short, I like my list of objects stored to be retrieved by a




  1. Retrieve a single item by an ID (property of the json object


  2. Paged lists without filters


  3. Paged lists with paging and filtering of json objects


  4. At any time any of the event objects can be changed by the user, so the cache needs to be updated (invalidate cache/update cache)



I am writing the code in .NET so if that makes any difference.







c# .net caching redis






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Dec 30 '18 at 20:26









ZoinkyZoinky

1,31132037




1,31132037













  • Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.

    – Nate
    Dec 30 '18 at 20:36











  • @nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds

    – Zoinky
    Dec 30 '18 at 20:41











  • If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.

    – Nate
    Dec 30 '18 at 20:49






  • 1





    Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.

    – Nate
    Dec 30 '18 at 20:50











  • @Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.

    – Zoinky
    Dec 30 '18 at 20:57





















  • Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.

    – Nate
    Dec 30 '18 at 20:36











  • @nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds

    – Zoinky
    Dec 30 '18 at 20:41











  • If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.

    – Nate
    Dec 30 '18 at 20:49






  • 1





    Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.

    – Nate
    Dec 30 '18 at 20:50











  • @Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.

    – Zoinky
    Dec 30 '18 at 20:57



















Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.

– Nate
Dec 30 '18 at 20:36





Are you sure Redis is the best tool for the job? Redis is primarily a key/value store. Filtering on values is not its strong suit by any means. Why are you wanting to use Redis for this over some sort of database that’s designed to do filtering and pagination? I wouldn’t personally use Redis for this, which is probably why you’re having trouble figuring out how to do it.

– Nate
Dec 30 '18 at 20:36













@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds

– Zoinky
Dec 30 '18 at 20:41





@nate I am already using a RDBM, I am introducing possibly using Redis to offload the work from the db server and make queries faster as its in memory retrieval vs having to go to the db. I am also questioning if the filtering part should be part of Redis, but I would like to at least get the paging and single item retrieval working if possible with reds

– Zoinky
Dec 30 '18 at 20:41













If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.

– Nate
Dec 30 '18 at 20:49





If I were you, I’d see if there is any performance tuning you can do to your database. Perhaps maybe you’re missing some key indices or something. This kind of stuff should be lightning fast even with an RDBM. If it’s not, you probably have a database problem that should be fixed instead of ignored. I know that doesn’t answer your question here, but I think it’s the best approach.

– Nate
Dec 30 '18 at 20:49




1




1





Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.

– Nate
Dec 30 '18 at 20:50





Also, I would examine the queries you’re running. Sometimes people run not very good queries and then are surprised when they’re slow.

– Nate
Dec 30 '18 at 20:50













@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.

– Zoinky
Dec 30 '18 at 20:57







@Nate Thanks for the replies, the queries are very lightweight and will be tuned to hell with indexes and such. The idea of introducing Redis is purely as another option if and when the tuning and indexes on the db are not good enough and Redis can be turned on and off. My past experiences says that in-memory (not Redis) cache can be lightening faster than the most tuned db in most cases but never done the actual paging/filtering against the in-mem objects. This is a new territory for me hence why I am exploding more options.

– Zoinky
Dec 30 '18 at 20:57














1 Answer
1






active

oldest

votes


















0














Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.



If I understand your requirements correctly, here is how I'd approach:




  1. Store objects in a sorted set (if you the returned objects in a specific order)


  2. You can now querying single and unfiltered objects by native Redis commands:
    Redis - Sorted set, find item by property value


  3. For filtered objects you can look into Lua scripts


  4. Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis



You can also check ignite, it has a couple of features built in that may be of interest to you:




  1. Binary marshaller :
    "It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
    https://apacheignite.readme.io/docs/binary-marshaller


  2. Read/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html







share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53981162%2fredis-which-datatype-would-be-most-efficient-way-to-store-data%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.



    If I understand your requirements correctly, here is how I'd approach:




    1. Store objects in a sorted set (if you the returned objects in a specific order)


    2. You can now querying single and unfiltered objects by native Redis commands:
      Redis - Sorted set, find item by property value


    3. For filtered objects you can look into Lua scripts


    4. Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis



    You can also check ignite, it has a couple of features built in that may be of interest to you:




    1. Binary marshaller :
      "It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
      https://apacheignite.readme.io/docs/binary-marshaller


    2. Read/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html







    share|improve this answer




























      0














      Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.



      If I understand your requirements correctly, here is how I'd approach:




      1. Store objects in a sorted set (if you the returned objects in a specific order)


      2. You can now querying single and unfiltered objects by native Redis commands:
        Redis - Sorted set, find item by property value


      3. For filtered objects you can look into Lua scripts


      4. Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis



      You can also check ignite, it has a couple of features built in that may be of interest to you:




      1. Binary marshaller :
        "It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
        https://apacheignite.readme.io/docs/binary-marshaller


      2. Read/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html







      share|improve this answer


























        0












        0








        0







        Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.



        If I understand your requirements correctly, here is how I'd approach:




        1. Store objects in a sorted set (if you the returned objects in a specific order)


        2. You can now querying single and unfiltered objects by native Redis commands:
          Redis - Sorted set, find item by property value


        3. For filtered objects you can look into Lua scripts


        4. Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis



        You can also check ignite, it has a couple of features built in that may be of interest to you:




        1. Binary marshaller :
          "It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
          https://apacheignite.readme.io/docs/binary-marshaller


        2. Read/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html







        share|improve this answer













        Seems like on top of doing simple key value queries, what you need it some additional logic to run on the server (Redis) side. You can use Lua scripts in Redis to perform such tasks.



        If I understand your requirements correctly, here is how I'd approach:




        1. Store objects in a sorted set (if you the returned objects in a specific order)


        2. You can now querying single and unfiltered objects by native Redis commands:
          Redis - Sorted set, find item by property value


        3. For filtered objects you can look into Lua scripts


        4. Redis does not provide anything out of the box to invalidate/update cache stored in lists. And you'll have to write additional code to handle cache updation. Read more here: https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis



        You can also check ignite, it has a couple of features built in that may be of interest to you:




        1. Binary marshaller :
          "It enables you to read an arbitrary field from an object's serialized form without full object deserialization."
          https://apacheignite.readme.io/docs/binary-marshaller


        2. Read/Write through ability for cache updation: https://ignite.apache.org/use-cases/caching/database-caching.html








        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Dec 31 '18 at 18:24









        rainhackerrainhacker

        1038




        1038






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53981162%2fredis-which-datatype-would-be-most-efficient-way-to-store-data%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            7yIkzFwQ6,ZqD Jh6in4xb5,3A,mdcfjO3MqesbdUoG6qgCUmiEXQYt,HgZgzI2V s0oJX84DNhEDKbrSRSOkRmwsTm,wxhYguRxcBR3
            ZeSRRoc6P5,s vn VPhqSgsVq TB uLWBzo,M1SfuIxVPLZ nvzmVRQayYHS3IUii

            Popular posts from this blog

            Monofisismo

            Angular Downloading a file using contenturl with Basic Authentication

            Olmecas