DataReader running very slow












0














I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?



con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try



            For Each word As String In result

Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader

If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()

Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)

Next
FoundWords.Clear()
End Using
Loop

Else

Continue For

End If
End Using



End Using





Next

Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()

End Try
End Using
MsgBox("Complete!")


So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"



Any help or ideas would be greatly appreciated.










share|improve this question
























  • I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the ExecuteReader line, since that's the line that runs the query.
    – Damien_The_Unbeliever
    yesterday










  • So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
    – K.Madden
    yesterday






  • 1




    Yes, the methods named ExecuteXxx are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
    – Damien_The_Unbeliever
    yesterday












  • Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
    – K.Madden
    yesterday






  • 2




    @K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
    – Dan Guzman
    yesterday
















0














I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?



con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try



            For Each word As String In result

Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader

If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()

Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)

Next
FoundWords.Clear()
End Using
Loop

Else

Continue For

End If
End Using



End Using





Next

Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()

End Try
End Using
MsgBox("Complete!")


So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"



Any help or ideas would be greatly appreciated.










share|improve this question
























  • I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the ExecuteReader line, since that's the line that runs the query.
    – Damien_The_Unbeliever
    yesterday










  • So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
    – K.Madden
    yesterday






  • 1




    Yes, the methods named ExecuteXxx are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
    – Damien_The_Unbeliever
    yesterday












  • Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
    – K.Madden
    yesterday






  • 2




    @K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
    – Dan Guzman
    yesterday














0












0








0


1





I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?



con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try



            For Each word As String In result

Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader

If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()

Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)

Next
FoundWords.Clear()
End Using
Loop

Else

Continue For

End If
End Using



End Using





Next

Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()

End Try
End Using
MsgBox("Complete!")


So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"



Any help or ideas would be greatly appreciated.










share|improve this question















I'm currently working with a database of over 50 million records, where I read a file which a person wants to search the database for etc. I have noticed my data reader part is running particularly slow, where as the query seems almost instant (database is indexed). I was just wondering does anyone know as to why it seems to be running slow?



con.Open()
Using sw As New StreamWriter("G:USERSEARCH-RESULTS.txt")
Try



            For Each word As String In result

Using com As New SqlCommand("select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd, t.LnePos from (Select SmeNbr, MAX(FilDte) as MaxDate from Test_Table where SmeNbr = @word group by SmeNbr)r inner join Test_Table t on t.SmeNbr = r.SmeNbr and t.FilDte = R.MaxDate", con)
com.Parameters.AddWithValue("@word", word)
Using RDR = com.ExecuteReader

If RDR.HasRows Then
Do While RDR.Read
MyFilePath = RDR.Item("FilPth").ToString()
linePos = RDR.Item("LnePos").ToString()

Using sr As New StreamReader(MyFilePath)
sr.BaseStream.Seek(4096 * (linePos - 1), SeekOrigin.Begin)
FoundWords.Add(sr.ReadLine)
For Each item As String In FoundWords
sw.WriteLine(item)

Next
FoundWords.Clear()
End Using
Loop

Else

Continue For

End If
End Using



End Using





Next

Catch ex As Exception
MessageBox.Show("Couldn't process search")
Finally
con.Close()

End Try
End Using
MsgBox("Complete!")


So it works perfect, as in it gets the records and bits of info I want very quickly through the query and all and even the writing reults to a new file is near instant, I used breakpoints and like I said it seems to take ages between the "Using RDR = com.ExecuteReader" and "If RDR.HasRows Then"



Any help or ideas would be greatly appreciated.







sql sql-server vb.net ssms






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited yesterday









Damien_The_Unbeliever

192k17245331




192k17245331










asked yesterday









K.Madden

257




257












  • I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the ExecuteReader line, since that's the line that runs the query.
    – Damien_The_Unbeliever
    yesterday










  • So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
    – K.Madden
    yesterday






  • 1




    Yes, the methods named ExecuteXxx are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
    – Damien_The_Unbeliever
    yesterday












  • Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
    – K.Madden
    yesterday






  • 2




    @K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
    – Dan Guzman
    yesterday


















  • I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the ExecuteReader line, since that's the line that runs the query.
    – Damien_The_Unbeliever
    yesterday










  • So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
    – K.Madden
    yesterday






  • 1




    Yes, the methods named ExecuteXxx are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
    – Damien_The_Unbeliever
    yesterday












  • Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
    – K.Madden
    yesterday






  • 2




    @K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
    – Dan Guzman
    yesterday
















I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the ExecuteReader line, since that's the line that runs the query.
– Damien_The_Unbeliever
yesterday




I don't know what you mean by "the query seems almost instant" if you're indicating that the slowness is in executing the ExecuteReader line, since that's the line that runs the query.
– Damien_The_Unbeliever
yesterday












So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday




So would execute reader also execute my whole "select t.SmeNbr, t.FilPth, r.MaxDate, t.DteAdd,"......etc.?
– K.Madden
yesterday




1




1




Yes, the methods named ExecuteXxx are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
– Damien_The_Unbeliever
yesterday






Yes, the methods named ExecuteXxx are the ones that execute the queries. Anything before that (except opening the connection) is just local preparation not involving the server at all.
– Damien_The_Unbeliever
yesterday














Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday




Yes sorry I was being so stupid of course, then if you could help me understand as to why when I perform that query in MSSMS its near instant but it seems to take much long in VB to execute? Also is there a way to speed it up? Thank you btw! :)
– K.Madden
yesterday




2




2




@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday




@K.Madden, yes. The reason it runs faster in SSMS is likely because you used a literal instead of parameter. You would have had the same issue in SSMS if you specified a Unicode string literal (e.g. N'SmeNbrValue').
– Dan Guzman
yesterday












1 Answer
1






active

oldest

votes


















3















com.Parameters.AddWithValue("@word", word)




AddWithValue infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n) parameter with the length of the actual value. I see from your comments that the actual column data type is char(13) so it would be best to explicitly specify that as the parameter data type:



com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word


The implications with AddWithValue are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.






share|improve this answer





















  • Brilliant! Worked like a charm! Thank you so much. :D
    – K.Madden
    yesterday











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944526%2fdatareader-running-very-slow%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









3















com.Parameters.AddWithValue("@word", word)




AddWithValue infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n) parameter with the length of the actual value. I see from your comments that the actual column data type is char(13) so it would be best to explicitly specify that as the parameter data type:



com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word


The implications with AddWithValue are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.






share|improve this answer





















  • Brilliant! Worked like a charm! Thank you so much. :D
    – K.Madden
    yesterday
















3















com.Parameters.AddWithValue("@word", word)




AddWithValue infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n) parameter with the length of the actual value. I see from your comments that the actual column data type is char(13) so it would be best to explicitly specify that as the parameter data type:



com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word


The implications with AddWithValue are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.






share|improve this answer





















  • Brilliant! Worked like a charm! Thank you so much. :D
    – K.Madden
    yesterday














3












3








3







com.Parameters.AddWithValue("@word", word)




AddWithValue infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n) parameter with the length of the actual value. I see from your comments that the actual column data type is char(13) so it would be best to explicitly specify that as the parameter data type:



com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word


The implications with AddWithValue are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.






share|improve this answer













com.Parameters.AddWithValue("@word", word)




AddWithValue infers the parameter data type from the provided .NET object value. Since .NET strings are Unicode, this code is will add an nvarchar(n) parameter with the length of the actual value. I see from your comments that the actual column data type is char(13) so it would be best to explicitly specify that as the parameter data type:



com.Parameters.Add("@word", SqlDbType.Char, 13).Value = word


The implications with AddWithValue are that indexes might not be used due to the mismatched data type and there may be many variations of the same query in the SQL Server procedure cache that differ only by length. For these reasons, I suggest one avoid AddWithValue.







share|improve this answer












share|improve this answer



share|improve this answer










answered yesterday









Dan Guzman

22.9k31540




22.9k31540












  • Brilliant! Worked like a charm! Thank you so much. :D
    – K.Madden
    yesterday


















  • Brilliant! Worked like a charm! Thank you so much. :D
    – K.Madden
    yesterday
















Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday




Brilliant! Worked like a charm! Thank you so much. :D
– K.Madden
yesterday


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53944526%2fdatareader-running-very-slow%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Monofisismo

Angular Downloading a file using contenturl with Basic Authentication

Olmecas