How to Archive Storage tables?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I have a scenario, where daily a storage table will be created with current date.e.g. Table412019 . So after some interval i need to archive the tables , i.e. in each month i need to archive all the table or need to merge all tables into a single table named Jan2019( which will contain all table rows Table112019 to Table1312019).
What is the best way to do it. All are in the Azure Table storage account.
azure azure-storage azure-table-storage asked Jan 4 at 9:14
TitiTiti
307
add a comment |
I have a scenario, where daily a storage table will be created with current date.e.g. Table412019 . So after some interval i need to archive the tables , i.e. in each month i need to archive all the table or need to merge all tables into a single table named Jan2019( which will contain all table rows Table112019 to Table1312019).
What is the best way to do it. All are in the Azure Table storage account.
azure azure-storage azure-table-storage asked Jan 4 at 9:14
TitiTiti
307
What's the purpose of the archive? Are you looking for a way to archive in a cheaper manner? What have you tried so far? Where are you getting stuck? Give us some more (relevant) info/code so we can help. Please refer to How to Ask and try to provide a Minimal, Complete, and Verifiable example.
– rickvdbosch
Jan 4 at 9:17
You are right, we are trying to store the data for future analysis in cheaper manner. Where as i thought of Azure table copy option , or ADF . I haven't tried anything hand on. So there is no way that i can put some code
– Titi
Jan 4 at 9:19
I am not looking for a exact code to do it, rather i am looking for the approach that i can avail which is available & cheaper.
– Titi
Jan 4 at 9:20
add a comment |
I have a scenario, where daily a storage table will be created with current date.e.g. Table412019 . So after some interval i need to archive the tables , i.e. in each month i need to archive all the table or need to merge all tables into a single table named Jan2019( which will contain all table rows Table112019 to Table1312019).
What is the best way to do it. All are in the Azure Table storage account.
azure azure-storage azure-table-storage asked Jan 4 at 9:14
TitiTiti
307
I have a scenario, where daily a storage table will be created with current date.e.g. Table412019 . So after some interval i need to archive the tables , i.e. in each month i need to archive all the table or need to merge all tables into a single table named Jan2019( which will contain all table rows Table112019 to Table1312019).
What is the best way to do it. All are in the Azure Table storage account.
azure azure-storage azure-table-storage azure azure-storage azure-table-storage asked Jan 4 at 9:14
TitiTiti
307
asked Jan 4 at 9:14
TitiTiti
307
asked Jan 4 at 9:14
TitiTiti
307
asked Jan 4 at 9:14
TitiTiti
307
asked Jan 4 at 9:14
TitiTiti
307
307
What's the purpose of the archive? Are you looking for a way to archive in a cheaper manner? What have you tried so far? Where are you getting stuck? Give us some more (relevant) info/code so we can help. Please refer to How to Ask and try to provide a Minimal, Complete, and Verifiable example.
– rickvdbosch
Jan 4 at 9:17
You are right, we are trying to store the data for future analysis in cheaper manner. Where as i thought of Azure table copy option , or ADF . I haven't tried anything hand on. So there is no way that i can put some code
– Titi
Jan 4 at 9:19
I am not looking for a exact code to do it, rather i am looking for the approach that i can avail which is available & cheaper.
– Titi
Jan 4 at 9:20
add a comment |
What's the purpose of the archive? Are you looking for a way to archive in a cheaper manner? What have you tried so far? Where are you getting stuck? Give us some more (relevant) info/code so we can help. Please refer to How to Ask and try to provide a Minimal, Complete, and Verifiable example.
– rickvdbosch
Jan 4 at 9:17
You are right, we are trying to store the data for future analysis in cheaper manner. Where as i thought of Azure table copy option , or ADF . I haven't tried anything hand on. So there is no way that i can put some code
– Titi
Jan 4 at 9:19
I am not looking for a exact code to do it, rather i am looking for the approach that i can avail which is available & cheaper.
– Titi
Jan 4 at 9:20
What's the purpose of the archive? Are you looking for a way to archive in a cheaper manner? What have you tried so far? Where are you getting stuck? Give us some more (relevant) info/code so we can help. Please refer to How to Ask and try to provide a Minimal, Complete, and Verifiable example.
– rickvdbosch
Jan 4 at 9:17
What's the purpose of the archive? Are you looking for a way to archive in a cheaper manner? What have you tried so far? Where are you getting stuck? Give us some more (relevant) info/code so we can help. Please refer to How to Ask and try to provide a Minimal, Complete, and Verifiable example.
– rickvdbosch
Jan 4 at 9:17
You are right, we are trying to store the data for future analysis in cheaper manner. Where as i thought of Azure table copy option , or ADF . I haven't tried anything hand on. So there is no way that i can put some code
– Titi
Jan 4 at 9:19
You are right, we are trying to store the data for future analysis in cheaper manner. Where as i thought of Azure table copy option , or ADF . I haven't tried anything hand on. So there is no way that i can put some code
– Titi
Jan 4 at 9:19
I am not looking for a exact code to do it, rather i am looking for the approach that i can avail which is available & cheaper.
– Titi
Jan 4 at 9:20
I am not looking for a exact code to do it, rather i am looking for the approach that i can avail which is available & cheaper.
– Titi
Jan 4 at 9:20
add a comment |
2 Answers
2
active
oldest
votes
The Hot, Cool, and Archive storage tiers are only available for Blob Storage (Source: Azure Blob storage: Premium (preview), Hot, Cool, and Archive storage tiers).
Since you're talking about Table Storage, migrating the data to an 'archive' will not help with the costs.
Moving the data into 'just' another table within the same storage account should be easy enough. You can, for instance, use Azure Data Factory or AZCopy.
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
add a comment |
If you want to merge multiple tables into one single table, I hope my way could help you.I used logic app to implement it.
These two tables are prepared for merging.
After merging it would be like this:
And here is my Logic app flow:
Firstly get Entities from table and table2:
Secondly choose action "Insert or Merge Entity":
After these save it and Run Trigger.If you still have other questions, please let me know.
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
|
show 1 more comment
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54035961%2fhow-to-archive-storage-tables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
The Hot, Cool, and Archive storage tiers are only available for Blob Storage (Source: Azure Blob storage: Premium (preview), Hot, Cool, and Archive storage tiers).
Since you're talking about Table Storage, migrating the data to an 'archive' will not help with the costs.
Moving the data into 'just' another table within the same storage account should be easy enough. You can, for instance, use Azure Data Factory or AZCopy.
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
add a comment |
The Hot, Cool, and Archive storage tiers are only available for Blob Storage (Source: Azure Blob storage: Premium (preview), Hot, Cool, and Archive storage tiers).
Since you're talking about Table Storage, migrating the data to an 'archive' will not help with the costs.
Moving the data into 'just' another table within the same storage account should be easy enough. You can, for instance, use Azure Data Factory or AZCopy.
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
add a comment |
The Hot, Cool, and Archive storage tiers are only available for Blob Storage (Source: Azure Blob storage: Premium (preview), Hot, Cool, and Archive storage tiers).
Since you're talking about Table Storage, migrating the data to an 'archive' will not help with the costs.
Moving the data into 'just' another table within the same storage account should be easy enough. You can, for instance, use Azure Data Factory or AZCopy.
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
The Hot, Cool, and Archive storage tiers are only available for Blob Storage (Source: Azure Blob storage: Premium (preview), Hot, Cool, and Archive storage tiers).
Since you're talking about Table Storage, migrating the data to an 'archive' will not help with the costs.
Moving the data into 'just' another table within the same storage account should be easy enough. You can, for instance, use Azure Data Factory or AZCopy.
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
answered Jan 4 at 10:01
rickvdboschrickvdbosch
4,37121727
4,37121727
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
add a comment |
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
So can i use data factory to merge multiple storage table (form 1st jan to 31st jan tables) into a single table(assume Jan2019Table)?
– Titi
Jan 4 at 10:12
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
I tried AzCopy its working, but we need to transfer the table data to a blob then from there .manifest file will be helpful to restore into the single table storage. But its manual.. need to think of doing it in Powershell to Automate. If Azcopy is there in Powershell?
– Titi
Jan 4 at 10:14
add a comment |
If you want to merge multiple tables into one single table, I hope my way could help you.I used logic app to implement it.
These two tables are prepared for merging.
After merging it would be like this:
And here is my Logic app flow:
Firstly get Entities from table and table2:
Secondly choose action "Insert or Merge Entity":
After these save it and Run Trigger.If you still have other questions, please let me know.
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
|
show 1 more comment
If you want to merge multiple tables into one single table, I hope my way could help you.I used logic app to implement it.
These two tables are prepared for merging.
After merging it would be like this:
And here is my Logic app flow:
Firstly get Entities from table and table2:
Secondly choose action "Insert or Merge Entity":
After these save it and Run Trigger.If you still have other questions, please let me know.
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
|
show 1 more comment
If you want to merge multiple tables into one single table, I hope my way could help you.I used logic app to implement it.
These two tables are prepared for merging.
After merging it would be like this:
And here is my Logic app flow:
Firstly get Entities from table and table2:
Secondly choose action "Insert or Merge Entity":
After these save it and Run Trigger.If you still have other questions, please let me know.
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
If you want to merge multiple tables into one single table, I hope my way could help you.I used logic app to implement it.
These two tables are prepared for merging.
After merging it would be like this:
And here is my Logic app flow:
Firstly get Entities from table and table2:
Secondly choose action "Insert or Merge Entity":
After these save it and Run Trigger.If you still have other questions, please let me know.
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
answered Jan 7 at 8:22
George ChenGeorge Chen
1,719119
1,719119
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
|
show 1 more comment
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
Although this is a nice approach, this will be feasible when we have less number of tables , but in my case the table count will vary according to the days count/month..
– Titi
Jan 8 at 12:15
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
@Titi， per my understanding, you better change the way you store data. You could store your data with a "time" property in months. Because no matter which way you choose, as time goes on, your table count will be huge, even there is a service could merge multiple tables mechanically , there will still be a timeout problem.
– George Chen
Jan 9 at 7:36
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
Thought of doing it, but scenario is different, as per day the record count may exceed more then 10-20 lacs so single table is a good option so at least we can open the table , otherwise for huge data it crashes sometime ..
– Titi
Jan 11 at 7:32
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi，you mean you store data in moths, and store more than 10-20 entities everyday, then you table will crash? How could that be? The max size of table scale target is huge.
– George Chen
Jan 11 at 7:41
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
@Titi And when your table is huge, what do you mean "crash"? Could you check table with "Azure Storage Explorer" tool?
– George Chen
Jan 11 at 8:46
|
show 1 more comment
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54035961%2fhow-to-archive-storage-tables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What's the purpose of the archive? Are you looking for a way to archive in a cheaper manner? What have you tried so far? Where are you getting stuck? Give us some more (relevant) info/code so we can help. Please refer to How to Ask and try to provide a Minimal, Complete, and Verifiable example.
– rickvdbosch
Jan 4 at 9:17
You are right, we are trying to store the data for future analysis in cheaper manner. Where as i thought of Azure table copy option , or ADF . I haven't tried anything hand on. So there is no way that i can put some code
– Titi
Jan 4 at 9:19
I am not looking for a exact code to do it, rather i am looking for the approach that i can avail which is available & cheaper.
– Titi
Jan 4 at 9:20