Bdtjtk

I have a Cassandra table with 9 million records and my data size is 500 MB . I have a Solr cloud with 3 nodes(3 shards and 2 replicas)with three external Zookeeper ensemble. My Cassandra is a 1 node cluster. I am trying to index this table using Apache Solr but my query is getting timeout as soon as i am starting full import .

I am able to cqlsh and fetch records but i am failing in indexing it .
Here is my attached solr.log...

Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT * from counter.series Processing Document # 1

        at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69)

        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:318)

        at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:279)

        at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:54)

        at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)

        at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)

        at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:244)

        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)

        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)

        ... 5 more

Caused by: java.sql.SQLTransientConnectionException: TimedOutException()

        at org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStatement.java:189)

        at org.apache.cassandra.cql.jdbc.CassandraStatement.execute(CassandraStatement.java:205)

        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.executeStatement(JdbcDataSource.java:338)

        at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:313)

        ... 12 more

Caused by: TimedOutException()

        at org.apache.cassandra.thrift.Cassandra$execute_cql3_query_result.read(Cassandra.java:37865)

        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)

        at org.apache.cassandra.thrift.Cassandra$Client.recv_execute_cql3_query(Cassandra.java:1562)

        at org.apache.cassandra.thrift.Cassandra$Client.execute_cql3_query(Cassandra.java:1547)

        at org.apache.cassandra.cql.jdbc.CassandraConnection.execute(CassandraConnection.java:468)

        at org.apache.cassandra.cql.jdbc.CassandraConnection.execute(CassandraConnection.java:494)

        at org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStatement.java:164)

        ... 15 more

I want some help in indexing the table either batch wise or by using multiple threads . Any help or suggestion is welcomed..
db-data-config.xml:
<dataConfig>
<dataSource type="JdbcDataSource" driver="org.apache.cassandra.cql.jdbc.CassandraDriver" url="jdbc:cassandra://192.168.0.7:9160/counter" user="cassandra" password="cassandra" autoCommit="true" />
<document>
<entity name="counter" query="SELECT * from counter.series;" autoCommit="true">
<field column="serial" name="serial" />
<field column="random" name="random" />
<field column="remarks" name="remarks" />
<field column="timestamp" name="timestamp" />
</entity>
</document>
</dataConfig>

solrconfig.xml

`<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">

<lst name="defaults">

  <str name="config">db-data-config.xml</str>

</lst>

`

schema.xml
<field name="remarks" type="string" indexed="false" stored="false" required="false" />
<field name="serial" type="string" indexed="true" stored="true" required="true" />
<field name="random" type="string" indexed="false" stored="true" required="true" />
<field name="timestamp" type="string" indexed="false" stored="false" required="false" />

The problem most likely is the size of the payload of data sending to Solr. By default when no batchSize specified in JdbcDataSource it's got defaulted to 500. It looks like in your case it's too much. You should use smaller numbers or increase timeout settings on Solr side