Why all the coefficients except the first(intercept) are obtaining the value very close to zero(e^-17 or low)...
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I wrote the following code in python using the statsmodels package, to create OLS regression model. I tried the code with different data-sets and got the model with all the coefficients values near to zero except the first(intercept) coefficient. What could possibly be wrong with the code ?
data1 = pandas.concat([Y, X], axis = 1)
dta = lagmat2ds(data1, mxlg, trim='both', dropex=1)
dtaown = sm.add_constant(dta[:, 0:(mxlg + 1)], prepend = False)
dtajoint = sm.add_constant(dta[:, 0:], prepend = False)
res2down = sm.OLS(dta[:, 0], dtaown).fit()
res2djoint = sm.OLS(dta[:, 0], dtajoint).fit()
Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
python python-3.x linear-regression statsmodels
asked Jan 4 at 9:10
SushodhanSushodhan
517
add a comment |
I wrote the following code in python using the statsmodels package, to create OLS regression model. I tried the code with different data-sets and got the model with all the coefficients values near to zero except the first(intercept) coefficient. What could possibly be wrong with the code ?
data1 = pandas.concat([Y, X], axis = 1)
dta = lagmat2ds(data1, mxlg, trim='both', dropex=1)
dtaown = sm.add_constant(dta[:, 0:(mxlg + 1)], prepend = False)
dtajoint = sm.add_constant(dta[:, 0:], prepend = False)
res2down = sm.OLS(dta[:, 0], dtaown).fit()
res2djoint = sm.OLS(dta[:, 0], dtajoint).fit()
Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
python python-3.x linear-regression statsmodels
asked Jan 4 at 9:10
SushodhanSushodhan
517
1
What aresm,lagmat2ds? Also, post the first few lines of your data -Y|X1|X2...
– Mortz
Jan 4 at 9:14
@Mortz Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
– Sushodhan
Jan 4 at 9:21
add a comment |
I wrote the following code in python using the statsmodels package, to create OLS regression model. I tried the code with different data-sets and got the model with all the coefficients values near to zero except the first(intercept) coefficient. What could possibly be wrong with the code ?
data1 = pandas.concat([Y, X], axis = 1)
dta = lagmat2ds(data1, mxlg, trim='both', dropex=1)
dtaown = sm.add_constant(dta[:, 0:(mxlg + 1)], prepend = False)
dtajoint = sm.add_constant(dta[:, 0:], prepend = False)
res2down = sm.OLS(dta[:, 0], dtaown).fit()
res2djoint = sm.OLS(dta[:, 0], dtajoint).fit()
Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
python python-3.x linear-regression statsmodels
asked Jan 4 at 9:10
SushodhanSushodhan
517
I wrote the following code in python using the statsmodels package, to create OLS regression model. I tried the code with different data-sets and got the model with all the coefficients values near to zero except the first(intercept) coefficient. What could possibly be wrong with the code ?
data1 = pandas.concat([Y, X], axis = 1)
dta = lagmat2ds(data1, mxlg, trim='both', dropex=1)
dtaown = sm.add_constant(dta[:, 0:(mxlg + 1)], prepend = False)
dtajoint = sm.add_constant(dta[:, 0:], prepend = False)
res2down = sm.OLS(dta[:, 0], dtaown).fit()
res2djoint = sm.OLS(dta[:, 0], dtajoint).fit()
Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
python python-3.x linear-regression statsmodels
python python-3.x linear-regression statsmodels
asked Jan 4 at 9:10
SushodhanSushodhan
517
asked Jan 4 at 9:10
SushodhanSushodhan
517
edited Jan 4 at 9:21
Sushodhan
asked Jan 4 at 9:10
SushodhanSushodhan
517
asked Jan 4 at 9:10
SushodhanSushodhan
517
asked Jan 4 at 9:10
SushodhanSushodhan
517
517
1
What aresm,lagmat2ds? Also, post the first few lines of your data -Y|X1|X2...
– Mortz
Jan 4 at 9:14
@Mortz Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
– Sushodhan
Jan 4 at 9:21
add a comment |
1
What aresm,lagmat2ds? Also, post the first few lines of your data -Y|X1|X2...
– Mortz
Jan 4 at 9:14
@Mortz Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
– Sushodhan
Jan 4 at 9:21
1
1
What are
sm, lagmat2ds? Also, post the first few lines of your data - Y|X1|X2...– Mortz
Jan 4 at 9:14
What are
sm, lagmat2ds? Also, post the first few lines of your data - Y|X1|X2...– Mortz
Jan 4 at 9:14
@Mortz Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
– Sushodhan
Jan 4 at 9:21
@Mortz Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
– Sushodhan
Jan 4 at 9:21
add a comment |
1 Answer
1
active
oldest
votes
The way your data is structured - you are modeling Y vs Y|lag Y|constant. Note that the OLS documentation (https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html) states that -
No constant is added by the model unless you are using formulas.
So the first value that you see is not the intercept but the coefficient of fitting Y vs Y - which will be 1.0.
What you can try to check that you are getting sensible results is to exclude Y from the predictors like this -
res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
answered Jan 4 at 10:53
MortzMortz
848619
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
1
What you are doing right now is fine - just make sure that you excludeYitself from the independent variables. Your current regression model isY=f(Y, Lagged_Y, X). If you removeYfrom the right hand side and fitY=f(Lagged_Y, X)- you should get a meaningful regression
– Mortz
Jan 4 at 12:24
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
1
IfYis your first column and you want to use all other columns to regress - use the last line in the answer -res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
– Mortz
Jan 7 at 8:49
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54035907%2fwhy-all-the-coefficients-except-the-firstintercept-are-obtaining-the-value-ver%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The way your data is structured - you are modeling Y vs Y|lag Y|constant. Note that the OLS documentation (https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html) states that -
No constant is added by the model unless you are using formulas.
So the first value that you see is not the intercept but the coefficient of fitting Y vs Y - which will be 1.0.
What you can try to check that you are getting sensible results is to exclude Y from the predictors like this -
res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
answered Jan 4 at 10:53
MortzMortz
848619
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
1
What you are doing right now is fine - just make sure that you excludeYitself from the independent variables. Your current regression model isY=f(Y, Lagged_Y, X). If you removeYfrom the right hand side and fitY=f(Lagged_Y, X)- you should get a meaningful regression
– Mortz
Jan 4 at 12:24
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
1
IfYis your first column and you want to use all other columns to regress - use the last line in the answer -res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
– Mortz
Jan 7 at 8:49
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
add a comment |
The way your data is structured - you are modeling Y vs Y|lag Y|constant. Note that the OLS documentation (https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html) states that -
No constant is added by the model unless you are using formulas.
So the first value that you see is not the intercept but the coefficient of fitting Y vs Y - which will be 1.0.
What you can try to check that you are getting sensible results is to exclude Y from the predictors like this -
res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
answered Jan 4 at 10:53
MortzMortz
848619
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
1
What you are doing right now is fine - just make sure that you excludeYitself from the independent variables. Your current regression model isY=f(Y, Lagged_Y, X). If you removeYfrom the right hand side and fitY=f(Lagged_Y, X)- you should get a meaningful regression
– Mortz
Jan 4 at 12:24
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
1
IfYis your first column and you want to use all other columns to regress - use the last line in the answer -res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
– Mortz
Jan 7 at 8:49
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
add a comment |
The way your data is structured - you are modeling Y vs Y|lag Y|constant. Note that the OLS documentation (https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html) states that -
No constant is added by the model unless you are using formulas.
So the first value that you see is not the intercept but the coefficient of fitting Y vs Y - which will be 1.0.
What you can try to check that you are getting sensible results is to exclude Y from the predictors like this -
res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
answered Jan 4 at 10:53
MortzMortz
848619
The way your data is structured - you are modeling Y vs Y|lag Y|constant. Note that the OLS documentation (https://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.OLS.html) states that -
No constant is added by the model unless you are using formulas.
So the first value that you see is not the intercept but the coefficient of fitting Y vs Y - which will be 1.0.
What you can try to check that you are getting sensible results is to exclude Y from the predictors like this -
res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
answered Jan 4 at 10:53
MortzMortz
848619
answered Jan 4 at 10:53
MortzMortz
848619
answered Jan 4 at 10:53
MortzMortz
848619
answered Jan 4 at 10:53
MortzMortz
848619
848619
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
1
What you are doing right now is fine - just make sure that you excludeYitself from the independent variables. Your current regression model isY=f(Y, Lagged_Y, X). If you removeYfrom the right hand side and fitY=f(Lagged_Y, X)- you should get a meaningful regression
– Mortz
Jan 4 at 12:24
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
1
IfYis your first column and you want to use all other columns to regress - use the last line in the answer -res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
– Mortz
Jan 7 at 8:49
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
add a comment |
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
1
What you are doing right now is fine - just make sure that you excludeYitself from the independent variables. Your current regression model isY=f(Y, Lagged_Y, X). If you removeYfrom the right hand side and fitY=f(Lagged_Y, X)- you should get a meaningful regression
– Mortz
Jan 4 at 12:24
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
1
IfYis your first column and you want to use all other columns to regress - use the last line in the answer -res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()
– Mortz
Jan 7 at 8:49
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
Okay. Thank you so much for pointing that out. I will surely try the way you mentioned. But the thing is that I want to build regression model which will consider previous i.e lagged values of Y as well as X. How can this be done ? Can you help me with this ?
– Sushodhan
Jan 4 at 12:20
1
1
What you are doing right now is fine - just make sure that you exclude
Y itself from the independent variables. Your current regression model is Y=f(Y, Lagged_Y, X). If you remove Y from the right hand side and fit Y=f(Lagged_Y, X) - you should get a meaningful regression– Mortz
Jan 4 at 12:24
What you are doing right now is fine - just make sure that you exclude
Y itself from the independent variables. Your current regression model is Y=f(Y, Lagged_Y, X). If you remove Y from the right hand side and fit Y=f(Lagged_Y, X) - you should get a meaningful regression– Mortz
Jan 4 at 12:24
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
Okay. Now it is clear to me where the problem is and what should be done to solve this. Can you please help me once more ? Just tell me what should I write in python(the code) to do the required ?
– Sushodhan
Jan 7 at 5:43
1
1
If
Y is your first column and you want to use all other columns to regress - use the last line in the answer - res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()– Mortz
Jan 7 at 8:49
If
Y is your first column and you want to use all other columns to regress - use the last line in the answer - res2down = sm.OLS(dta[:, 0], dtaown[:, 1:]).fit()– Mortz
Jan 7 at 8:49
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
Thank you very much. It worked. You were a great help to me. :)
– Sushodhan
Jan 7 at 9:14
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54035907%2fwhy-all-the-coefficients-except-the-firstintercept-are-obtaining-the-value-ver%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
What are
sm,lagmat2ds? Also, post the first few lines of your data -Y|X1|X2...– Mortz
Jan 4 at 9:14
@Mortz Here the sm is statsmodels.api as sm and for sample testing you can consider the dataset sm.datasets.spector.
– Sushodhan
Jan 4 at 9:21