Error check/handle missing values in web API call
I'm working on an application that retrieves weather data from a server. The server is updated every 5 minutes with weather data from various weather stations. The issue is that if there is a connection problem, or the station is unplugged there ends up being missing values. The API returns the timestamp specified, the time step between measurements, and a list of measurements.
As an example if I try to get a value every hour for a given day (24hr period) I would expect 24 values (168 for a week and 720 for 30/744 for 31days). If there's measurements missing the result is the same minus the missing values and I no way of knowing if their missing from the beginning, middle or end of the "value" key.
I have tried smaller time intervals (every value for a period - day, week, month) in order to fill the gaps between possible missing measurements but the API has request limits including a size limit that renders the smaller interval unpractical for larger requests (such as a month).
example of what is returned from the server-
{
"beg_time"= 1533097205,
"step_time": 3600,
"value":[
[14.7],
[14.6],
[14.5],
...
[14.8]]
}
So if I end up missing measurements and end up with less than expected how would I begin to error check that? The desired results is to input the data into a file with a timestamp for each measurement. Missing values are okay but I need to know how to find them in the list. I can only relate the beg_time to the first value and each subsequent value has a time of beg_time + step_time. I need to be able to know where I'm missing a value so I can still add the timestamp into the file but leave the measurement blank.
Full example of incomplete response: only 23 values even though 24 is expected.
[
{
"beg_time": 1533009600,
"step_time": 3600,
"value": [
[
14.6
],
[
15.1
],
[
14.2
],
[
12.8
],
[
11.6
],
[
10.7
],
[
11.5
],
[
16.3
],
[
22.3
],
[
28.2
],
[
29.4
],
[
29.1
],
[
29.3
],
[
29.6
],
[
30.5
],
[
29.7
],
[
28.9
],
[
26.3
],
[
24.8
],
[
23.6
],
[
21.2
],
[
17.9
],
[
15.9
]
]
}
]
python json api parsing
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
|
show 6 more comments
I'm working on an application that retrieves weather data from a server. The server is updated every 5 minutes with weather data from various weather stations. The issue is that if there is a connection problem, or the station is unplugged there ends up being missing values. The API returns the timestamp specified, the time step between measurements, and a list of measurements.
As an example if I try to get a value every hour for a given day (24hr period) I would expect 24 values (168 for a week and 720 for 30/744 for 31days). If there's measurements missing the result is the same minus the missing values and I no way of knowing if their missing from the beginning, middle or end of the "value" key.
I have tried smaller time intervals (every value for a period - day, week, month) in order to fill the gaps between possible missing measurements but the API has request limits including a size limit that renders the smaller interval unpractical for larger requests (such as a month).
example of what is returned from the server-
{
"beg_time"= 1533097205,
"step_time": 3600,
"value":[
[14.7],
[14.6],
[14.5],
...
[14.8]]
}
So if I end up missing measurements and end up with less than expected how would I begin to error check that? The desired results is to input the data into a file with a timestamp for each measurement. Missing values are okay but I need to know how to find them in the list. I can only relate the beg_time to the first value and each subsequent value has a time of beg_time + step_time. I need to be able to know where I'm missing a value so I can still add the timestamp into the file but leave the measurement blank.
Full example of incomplete response: only 23 values even though 24 is expected.
[
{
"beg_time": 1533009600,
"step_time": 3600,
"value": [
[
14.6
],
[
15.1
],
[
14.2
],
[
12.8
],
[
11.6
],
[
10.7
],
[
11.5
],
[
16.3
],
[
22.3
],
[
28.2
],
[
29.4
],
[
29.1
],
[
29.3
],
[
29.6
],
[
30.5
],
[
29.7
],
[
28.9
],
[
26.3
],
[
24.8
],
[
23.6
],
[
21.2
],
[
17.9
],
[
15.9
]
]
}
]
python json api parsing
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
You don't know it's missing because it's not there. Is that the question? You have to save something when you can't get the measurement. Maybe your problem is that the reading code is blocking and if the reading does not come, your code is still blocked until the next read?
– progmatico
Dec 28 '18 at 23:25
When you get an error you should save something in the database that indicates that you didn't get a measurement at that time. Then you can put an empty value in the response for that step,[[14.7], [14.6], , [14.5], ...]
– Barmar
Dec 28 '18 at 23:27
Exactly. Like if I only get 23 values back when I expect 24 how do I know where I should be putting the empty time value in the file. The issue is just the the value at that specified time (hour increments) may or may not be missing. Its generally due to stations disconnecting (being unplugged or power outages) or wifi connection issues. Most of the weather stations are in the far northern communities of Ontario Canada, and they can have spotty power/wifi issues sometimes. I want to ensure that when they use my program it is able to account for possible missing values.
– Seraphiim
Dec 28 '18 at 23:31
@Barmar the API and Server are not accessible and are third party
– Seraphiim
Dec 28 '18 at 23:33
1
@user2340612 I am using that API and for the interval that I am using 1hour it does not separate into different objects per timestamp. I have noticed it do that for "max" but when Im asking for a range of a month the API limits me.
– Seraphiim
Dec 28 '18 at 23:59
|
show 6 more comments
I'm working on an application that retrieves weather data from a server. The server is updated every 5 minutes with weather data from various weather stations. The issue is that if there is a connection problem, or the station is unplugged there ends up being missing values. The API returns the timestamp specified, the time step between measurements, and a list of measurements.
As an example if I try to get a value every hour for a given day (24hr period) I would expect 24 values (168 for a week and 720 for 30/744 for 31days). If there's measurements missing the result is the same minus the missing values and I no way of knowing if their missing from the beginning, middle or end of the "value" key.
I have tried smaller time intervals (every value for a period - day, week, month) in order to fill the gaps between possible missing measurements but the API has request limits including a size limit that renders the smaller interval unpractical for larger requests (such as a month).
example of what is returned from the server-
{
"beg_time"= 1533097205,
"step_time": 3600,
"value":[
[14.7],
[14.6],
[14.5],
...
[14.8]]
}
So if I end up missing measurements and end up with less than expected how would I begin to error check that? The desired results is to input the data into a file with a timestamp for each measurement. Missing values are okay but I need to know how to find them in the list. I can only relate the beg_time to the first value and each subsequent value has a time of beg_time + step_time. I need to be able to know where I'm missing a value so I can still add the timestamp into the file but leave the measurement blank.
Full example of incomplete response: only 23 values even though 24 is expected.
[
{
"beg_time": 1533009600,
"step_time": 3600,
"value": [
[
14.6
],
[
15.1
],
[
14.2
],
[
12.8
],
[
11.6
],
[
10.7
],
[
11.5
],
[
16.3
],
[
22.3
],
[
28.2
],
[
29.4
],
[
29.1
],
[
29.3
],
[
29.6
],
[
30.5
],
[
29.7
],
[
28.9
],
[
26.3
],
[
24.8
],
[
23.6
],
[
21.2
],
[
17.9
],
[
15.9
]
]
}
]
python json api parsing
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
I'm working on an application that retrieves weather data from a server. The server is updated every 5 minutes with weather data from various weather stations. The issue is that if there is a connection problem, or the station is unplugged there ends up being missing values. The API returns the timestamp specified, the time step between measurements, and a list of measurements.
As an example if I try to get a value every hour for a given day (24hr period) I would expect 24 values (168 for a week and 720 for 30/744 for 31days). If there's measurements missing the result is the same minus the missing values and I no way of knowing if their missing from the beginning, middle or end of the "value" key.
I have tried smaller time intervals (every value for a period - day, week, month) in order to fill the gaps between possible missing measurements but the API has request limits including a size limit that renders the smaller interval unpractical for larger requests (such as a month).
example of what is returned from the server-
{
"beg_time"= 1533097205,
"step_time": 3600,
"value":[
[14.7],
[14.6],
[14.5],
...
[14.8]]
}
So if I end up missing measurements and end up with less than expected how would I begin to error check that? The desired results is to input the data into a file with a timestamp for each measurement. Missing values are okay but I need to know how to find them in the list. I can only relate the beg_time to the first value and each subsequent value has a time of beg_time + step_time. I need to be able to know where I'm missing a value so I can still add the timestamp into the file but leave the measurement blank.
Full example of incomplete response: only 23 values even though 24 is expected.
[
{
"beg_time": 1533009600,
"step_time": 3600,
"value": [
[
14.6
],
[
15.1
],
[
14.2
],
[
12.8
],
[
11.6
],
[
10.7
],
[
11.5
],
[
16.3
],
[
22.3
],
[
28.2
],
[
29.4
],
[
29.1
],
[
29.3
],
[
29.6
],
[
30.5
],
[
29.7
],
[
28.9
],
[
26.3
],
[
24.8
],
[
23.6
],
[
21.2
],
[
17.9
],
[
15.9
]
]
}
]
python json api parsing
python json api parsing
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
edited Dec 28 '18 at 23:42
Seraphiim
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
asked Dec 28 '18 at 23:15
SeraphiimSeraphiim
32
32
You don't know it's missing because it's not there. Is that the question? You have to save something when you can't get the measurement. Maybe your problem is that the reading code is blocking and if the reading does not come, your code is still blocked until the next read?
– progmatico
Dec 28 '18 at 23:25
When you get an error you should save something in the database that indicates that you didn't get a measurement at that time. Then you can put an empty value in the response for that step,[[14.7], [14.6], , [14.5], ...]
– Barmar
Dec 28 '18 at 23:27
Exactly. Like if I only get 23 values back when I expect 24 how do I know where I should be putting the empty time value in the file. The issue is just the the value at that specified time (hour increments) may or may not be missing. Its generally due to stations disconnecting (being unplugged or power outages) or wifi connection issues. Most of the weather stations are in the far northern communities of Ontario Canada, and they can have spotty power/wifi issues sometimes. I want to ensure that when they use my program it is able to account for possible missing values.
– Seraphiim
Dec 28 '18 at 23:31
@Barmar the API and Server are not accessible and are third party
– Seraphiim
Dec 28 '18 at 23:33
1
@user2340612 I am using that API and for the interval that I am using 1hour it does not separate into different objects per timestamp. I have noticed it do that for "max" but when Im asking for a range of a month the API limits me.
– Seraphiim
Dec 28 '18 at 23:59
|
show 6 more comments
You don't know it's missing because it's not there. Is that the question? You have to save something when you can't get the measurement. Maybe your problem is that the reading code is blocking and if the reading does not come, your code is still blocked until the next read?
– progmatico
Dec 28 '18 at 23:25
When you get an error you should save something in the database that indicates that you didn't get a measurement at that time. Then you can put an empty value in the response for that step,[[14.7], [14.6], , [14.5], ...]
– Barmar
Dec 28 '18 at 23:27
Exactly. Like if I only get 23 values back when I expect 24 how do I know where I should be putting the empty time value in the file. The issue is just the the value at that specified time (hour increments) may or may not be missing. Its generally due to stations disconnecting (being unplugged or power outages) or wifi connection issues. Most of the weather stations are in the far northern communities of Ontario Canada, and they can have spotty power/wifi issues sometimes. I want to ensure that when they use my program it is able to account for possible missing values.
– Seraphiim
Dec 28 '18 at 23:31
@Barmar the API and Server are not accessible and are third party
– Seraphiim
Dec 28 '18 at 23:33
1
@user2340612 I am using that API and for the interval that I am using 1hour it does not separate into different objects per timestamp. I have noticed it do that for "max" but when Im asking for a range of a month the API limits me.
– Seraphiim
Dec 28 '18 at 23:59
You don't know it's missing because it's not there. Is that the question? You have to save something when you can't get the measurement. Maybe your problem is that the reading code is blocking and if the reading does not come, your code is still blocked until the next read?
– progmatico
Dec 28 '18 at 23:25
You don't know it's missing because it's not there. Is that the question? You have to save something when you can't get the measurement. Maybe your problem is that the reading code is blocking and if the reading does not come, your code is still blocked until the next read?
– progmatico
Dec 28 '18 at 23:25
When you get an error you should save something in the database that indicates that you didn't get a measurement at that time. Then you can put an empty value in the response for that step,
[[14.7], [14.6], , [14.5], ...]– Barmar
Dec 28 '18 at 23:27
When you get an error you should save something in the database that indicates that you didn't get a measurement at that time. Then you can put an empty value in the response for that step,
[[14.7], [14.6], , [14.5], ...]– Barmar
Dec 28 '18 at 23:27
Exactly. Like if I only get 23 values back when I expect 24 how do I know where I should be putting the empty time value in the file. The issue is just the the value at that specified time (hour increments) may or may not be missing. Its generally due to stations disconnecting (being unplugged or power outages) or wifi connection issues. Most of the weather stations are in the far northern communities of Ontario Canada, and they can have spotty power/wifi issues sometimes. I want to ensure that when they use my program it is able to account for possible missing values.
– Seraphiim
Dec 28 '18 at 23:31
Exactly. Like if I only get 23 values back when I expect 24 how do I know where I should be putting the empty time value in the file. The issue is just the the value at that specified time (hour increments) may or may not be missing. Its generally due to stations disconnecting (being unplugged or power outages) or wifi connection issues. Most of the weather stations are in the far northern communities of Ontario Canada, and they can have spotty power/wifi issues sometimes. I want to ensure that when they use my program it is able to account for possible missing values.
– Seraphiim
Dec 28 '18 at 23:31
@Barmar the API and Server are not accessible and are third party
– Seraphiim
Dec 28 '18 at 23:33
@Barmar the API and Server are not accessible and are third party
– Seraphiim
Dec 28 '18 at 23:33
1
1
@user2340612 I am using that API and for the interval that I am using 1hour it does not separate into different objects per timestamp. I have noticed it do that for "max" but when Im asking for a range of a month the API limits me.
– Seraphiim
Dec 28 '18 at 23:59
@user2340612 I am using that API and for the interval that I am using 1hour it does not separate into different objects per timestamp. I have noticed it do that for "max" but when Im asking for a range of a month the API limits me.
– Seraphiim
Dec 28 '18 at 23:59
|
show 6 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53965302%2ferror-check-handle-missing-values-in-web-api-call%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53965302%2ferror-check-handle-missing-values-in-web-api-call%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You don't know it's missing because it's not there. Is that the question? You have to save something when you can't get the measurement. Maybe your problem is that the reading code is blocking and if the reading does not come, your code is still blocked until the next read?
– progmatico
Dec 28 '18 at 23:25
When you get an error you should save something in the database that indicates that you didn't get a measurement at that time. Then you can put an empty value in the response for that step,
[[14.7], [14.6], , [14.5], ...]– Barmar
Dec 28 '18 at 23:27
Exactly. Like if I only get 23 values back when I expect 24 how do I know where I should be putting the empty time value in the file. The issue is just the the value at that specified time (hour increments) may or may not be missing. Its generally due to stations disconnecting (being unplugged or power outages) or wifi connection issues. Most of the weather stations are in the far northern communities of Ontario Canada, and they can have spotty power/wifi issues sometimes. I want to ensure that when they use my program it is able to account for possible missing values.
– Seraphiim
Dec 28 '18 at 23:31
@Barmar the API and Server are not accessible and are third party
– Seraphiim
Dec 28 '18 at 23:33
1
@user2340612 I am using that API and for the interval that I am using 1hour it does not separate into different objects per timestamp. I have noticed it do that for "max" but when Im asking for a range of a month the API limits me.
– Seraphiim
Dec 28 '18 at 23:59