DataReader running very slow
I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?
con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try
For Each word As String In result
Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader
If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()
Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)
Next
FoundWords.Clear()
End Using
Loop
Else
Continue For
End If
End Using
End Using
Next
Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()
End Try
End Using
MsgBox("Complete!")
So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"
Any help or ideas would be greatly appreciated.
sql sql-server vb.net ssms
|
show 2 more comments
I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?
con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try
For Each word As String In result
Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader
If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()
Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)
Next
FoundWords.Clear()
End Using
Loop
Else
Continue For
End If
End Using
End Using
Next
Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()
End Try
End Using
MsgBox("Complete!")
So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"
Any help or ideas would be greatly appreciated.
sql sql-server vb.net ssms
I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing theExecuteReader
line, since that's the line that runs the query.
– Damien_The_Unbeliever
yesterday
So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday
1
Yes, the methods namedExecuteXxx
are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
– Damien_The_Unbeliever
yesterday
Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday
2
@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday
|
show 2 more comments
I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?
con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try
For Each word As String In result
Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader
If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()
Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)
Next
FoundWords.Clear()
End Using
Loop
Else
Continue For
End If
End Using
End Using
Next
Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()
End Try
End Using
MsgBox("Complete!")
So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"
Any help or ideas would be greatly appreciated.
sql sql-server vb.net ssms
I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?
con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try
For Each word As String In result
Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader
If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()
Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)
Next
FoundWords.Clear()
End Using
Loop
Else
Continue For
End If
End Using
End Using
Next
Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()
End Try
End Using
MsgBox("Complete!")
So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"
Any help or ideas would be greatly appreciated.
sql sql-server vb.net ssms
sql sql-server vb.net ssms
edited yesterday
Damien_The_Unbeliever
192k17245331
192k17245331
asked yesterday
K.Madden
257
257
I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing theExecuteReader
line, since that's the line that runs the query.
– Damien_The_Unbeliever
yesterday
So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday
1
Yes, the methods namedExecuteXxx
are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
– Damien_The_Unbeliever
yesterday
Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday
2
@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday
|
show 2 more comments
I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing theExecuteReader
line, since that's the line that runs the query.
– Damien_The_Unbeliever
yesterday
So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday
1
Yes, the methods namedExecuteXxx
are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
– Damien_The_Unbeliever
yesterday
Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday
2
@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday
I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the
ExecuteReader
line, since that's the line that runs the query.– Damien_The_Unbeliever
yesterday
I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the
ExecuteReader
line, since that's the line that runs the query.– Damien_The_Unbeliever
yesterday
So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday
So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday
1
1
Yes, the methods named
ExecuteXxx
are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.– Damien_The_Unbeliever
yesterday
Yes, the methods named
ExecuteXxx
are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.– Damien_The_Unbeliever
yesterday
Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday
Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday
2
2
@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday
@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday
|
show 2 more comments
1 Answer
1
active
oldest
votes
com.Parameters.AddWithValue("@word", word)
AddWithValue
infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n)
parameter with the length of the actual value. I see from your comments that the actual column data type is char(13)
so it would be best to explicitly specify that as the parameter data type:
com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word
The implications with AddWithValue
are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944526%2fdatareader-running-very-slow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
com.Parameters.AddWithValue("@word", word)
AddWithValue
infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n)
parameter with the length of the actual value. I see from your comments that the actual column data type is char(13)
so it would be best to explicitly specify that as the parameter data type:
com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word
The implications with AddWithValue
are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
add a comment |
com.Parameters.AddWithValue("@word", word)
AddWithValue
infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n)
parameter with the length of the actual value. I see from your comments that the actual column data type is char(13)
so it would be best to explicitly specify that as the parameter data type:
com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word
The implications with AddWithValue
are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
add a comment |
com.Parameters.AddWithValue("@word", word)
AddWithValue
infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n)
parameter with the length of the actual value. I see from your comments that the actual column data type is char(13)
so it would be best to explicitly specify that as the parameter data type:
com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word
The implications with AddWithValue
are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.
com.Parameters.AddWithValue("@word", word)
AddWithValue
infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n)
parameter with the length of the actual value. I see from your comments that the actual column data type is char(13)
so it would be best to explicitly specify that as the parameter data type:
com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word
The implications with AddWithValue
are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.
answered yesterday
Dan Guzman
22.9k31540
22.9k31540
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
add a comment |
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944526%2fdatareader-running-very-slow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the
ExecuteReader
line, since that's the line that runs the query.– Damien_The_Unbeliever
yesterday
So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday
1
Yes, the methods named
ExecuteXxx
are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.– Damien_The_Unbeliever
yesterday
Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday
2
@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday