Apache Phoenix IndexTool failing with java.lang.ClassNotFoundException:...
0
I have Cloudera CDH 5.14.2 cluster with Apache Phoenix Parcel installed (APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3).
I have a table containing secondary index and I would like to populate this index using IndexTool provided with Apache Phoenix. But this is giving me the following error:
19/01/02 13:58:10 INFO mapreduce.Job: The url to track the job: http://mor-master-01.triviadata.local:8088/proxy/application_1546422102410_0020/
19/01/02 13:58:10 INFO mapreduce.Job: Running job: job_1546422102410_0020
19/01/02 13:58:18 INFO mapreduce.Job: Job job_1546422102410_0020 running in uber mode : false
19/01/02 13:58:18 INFO mapreduce.Job: map 0% reduce 0%
19/01/02 13:58:22 INFO mapreduce.Job: Task Id : attempt_1546422102410_0020_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.tephra.TransactionSystemClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.phoenix.transaction.TransactionFactory$Provider.<clinit>(TransactionFactory.java:27)
at org.apache.phoenix.query.QueryServicesOptions.<clinit>(QueryServicesOptions.java:270)
at org.apache.phoenix.query.QueryServicesImpl.<init>(QueryServicesImpl.java:36)
at org.apache.phoenix.jdbc.PhoenixDriver.getQueryServices(PhoenixDriver.java:197)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:235)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:113)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:58)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:180)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.createRecordReader(PhoenixInputFormat.java:76)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:521)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
When I check my HBASE_CLASSPATH with command ${HBASE_HOME}/bin/hbase classpath, I see it contains the following jars:
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/conf
/usr/java/latest/lib/tools.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/*.jar
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/hadoop/.//*
$HBASE_HOME/lib/hadoop-hdfs/./
$HBASE_HOME/hadoop-hdfs/lib/*
$HBASE_HOME/hadoop-hdfs/.//*
$HBASE_HOME/hadoop-yarn/lib/*
$HBASE_HOME/hadoop-yarn/.//*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/*
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/zookeeper/*
$HBASE_HOME/lib/zookeeper/lib/*
$HBASE_HOME/jars/hbase-common-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-client-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/htrace-core-3.0.4.jar
$HBASE_HOME/jars/htrace-core-3.2.0-incubating.jar
$HBASE_HOME/jars/htrace-core4-4.0.1-incubating.jar
$HBASE_HOME/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-server-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/metrics-core-2.2.0.jar
$HBASE_HOME/jars/metrics-core-3.1.2.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/joda-time-1.6.jar
$PHOENIX_HOME/lib/phoenix/lib/antlr-runtime-3.5.2.jar
When I check the source code and it's dependencies I see that the missing class is part of the $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
When I grep the content of this JAR for the missing class I see that it is there:
# jar tf $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar | grep TransactionSystemClient
org/apache/tephra/TransactionSystemClient.class
Do you have an idea why MR job cannot find this particular class?
If it helps, my table with secondary index is defined as follows:
0: jdbc:phoenix:localhost:2181/hbase> create table t1(v1 varchar, v2 varchar, v3 integer constraint primary_key primary key(v1)) immutable_rows=true, compression='SNAPPY';
1: jdbc:phoenix:localhost:2181/hbase> create index glb_idx on t1(v2) async;
And I run IndexTool with the command
${HBASE_HOME}/bin/hbase org.apache.phoenix.mapreduce.index.IndexTool -dt T1 -it GLB_IDX -op /tmp
When I create index synchronously and upsert some data into the table, index is populated correctly, so Phoenix secondary index configuration looks OK.
java cloudera-cdh phoenix
asked Jan 2 at 14:47
belobelo
587
add a comment |
0
I have Cloudera CDH 5.14.2 cluster with Apache Phoenix Parcel installed (APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3).
I have a table containing secondary index and I would like to populate this index using IndexTool provided with Apache Phoenix. But this is giving me the following error:
19/01/02 13:58:10 INFO mapreduce.Job: The url to track the job: http://mor-master-01.triviadata.local:8088/proxy/application_1546422102410_0020/
19/01/02 13:58:10 INFO mapreduce.Job: Running job: job_1546422102410_0020
19/01/02 13:58:18 INFO mapreduce.Job: Job job_1546422102410_0020 running in uber mode : false
19/01/02 13:58:18 INFO mapreduce.Job: map 0% reduce 0%
19/01/02 13:58:22 INFO mapreduce.Job: Task Id : attempt_1546422102410_0020_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.tephra.TransactionSystemClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.phoenix.transaction.TransactionFactory$Provider.<clinit>(TransactionFactory.java:27)
at org.apache.phoenix.query.QueryServicesOptions.<clinit>(QueryServicesOptions.java:270)
at org.apache.phoenix.query.QueryServicesImpl.<init>(QueryServicesImpl.java:36)
at org.apache.phoenix.jdbc.PhoenixDriver.getQueryServices(PhoenixDriver.java:197)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:235)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:113)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:58)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:180)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.createRecordReader(PhoenixInputFormat.java:76)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:521)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
When I check my HBASE_CLASSPATH with command ${HBASE_HOME}/bin/hbase classpath, I see it contains the following jars:
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/conf
/usr/java/latest/lib/tools.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/*.jar
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/hadoop/.//*
$HBASE_HOME/lib/hadoop-hdfs/./
$HBASE_HOME/hadoop-hdfs/lib/*
$HBASE_HOME/hadoop-hdfs/.//*
$HBASE_HOME/hadoop-yarn/lib/*
$HBASE_HOME/hadoop-yarn/.//*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/*
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/zookeeper/*
$HBASE_HOME/lib/zookeeper/lib/*
$HBASE_HOME/jars/hbase-common-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-client-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/htrace-core-3.0.4.jar
$HBASE_HOME/jars/htrace-core-3.2.0-incubating.jar
$HBASE_HOME/jars/htrace-core4-4.0.1-incubating.jar
$HBASE_HOME/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-server-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/metrics-core-2.2.0.jar
$HBASE_HOME/jars/metrics-core-3.1.2.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/joda-time-1.6.jar
$PHOENIX_HOME/lib/phoenix/lib/antlr-runtime-3.5.2.jar
When I check the source code and it's dependencies I see that the missing class is part of the $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
When I grep the content of this JAR for the missing class I see that it is there:
# jar tf $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar | grep TransactionSystemClient
org/apache/tephra/TransactionSystemClient.class
Do you have an idea why MR job cannot find this particular class?
If it helps, my table with secondary index is defined as follows:
0: jdbc:phoenix:localhost:2181/hbase> create table t1(v1 varchar, v2 varchar, v3 integer constraint primary_key primary key(v1)) immutable_rows=true, compression='SNAPPY';
1: jdbc:phoenix:localhost:2181/hbase> create index glb_idx on t1(v2) async;
And I run IndexTool with the command
${HBASE_HOME}/bin/hbase org.apache.phoenix.mapreduce.index.IndexTool -dt T1 -it GLB_IDX -op /tmp
When I create index synchronously and upsert some data into the table, index is populated correctly, so Phoenix secondary index configuration looks OK.
java cloudera-cdh phoenix
asked Jan 2 at 14:47
belobelo
587
add a comment |
0
0
0
I have Cloudera CDH 5.14.2 cluster with Apache Phoenix Parcel installed (APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3).
I have a table containing secondary index and I would like to populate this index using IndexTool provided with Apache Phoenix. But this is giving me the following error:
19/01/02 13:58:10 INFO mapreduce.Job: The url to track the job: http://mor-master-01.triviadata.local:8088/proxy/application_1546422102410_0020/
19/01/02 13:58:10 INFO mapreduce.Job: Running job: job_1546422102410_0020
19/01/02 13:58:18 INFO mapreduce.Job: Job job_1546422102410_0020 running in uber mode : false
19/01/02 13:58:18 INFO mapreduce.Job: map 0% reduce 0%
19/01/02 13:58:22 INFO mapreduce.Job: Task Id : attempt_1546422102410_0020_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.tephra.TransactionSystemClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.phoenix.transaction.TransactionFactory$Provider.<clinit>(TransactionFactory.java:27)
at org.apache.phoenix.query.QueryServicesOptions.<clinit>(QueryServicesOptions.java:270)
at org.apache.phoenix.query.QueryServicesImpl.<init>(QueryServicesImpl.java:36)
at org.apache.phoenix.jdbc.PhoenixDriver.getQueryServices(PhoenixDriver.java:197)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:235)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:113)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:58)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:180)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.createRecordReader(PhoenixInputFormat.java:76)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:521)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
When I check my HBASE_CLASSPATH with command ${HBASE_HOME}/bin/hbase classpath, I see it contains the following jars:
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/conf
/usr/java/latest/lib/tools.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/*.jar
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/hadoop/.//*
$HBASE_HOME/lib/hadoop-hdfs/./
$HBASE_HOME/hadoop-hdfs/lib/*
$HBASE_HOME/hadoop-hdfs/.//*
$HBASE_HOME/hadoop-yarn/lib/*
$HBASE_HOME/hadoop-yarn/.//*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/*
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/zookeeper/*
$HBASE_HOME/lib/zookeeper/lib/*
$HBASE_HOME/jars/hbase-common-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-client-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/htrace-core-3.0.4.jar
$HBASE_HOME/jars/htrace-core-3.2.0-incubating.jar
$HBASE_HOME/jars/htrace-core4-4.0.1-incubating.jar
$HBASE_HOME/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-server-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/metrics-core-2.2.0.jar
$HBASE_HOME/jars/metrics-core-3.1.2.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/joda-time-1.6.jar
$PHOENIX_HOME/lib/phoenix/lib/antlr-runtime-3.5.2.jar
When I check the source code and it's dependencies I see that the missing class is part of the $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
When I grep the content of this JAR for the missing class I see that it is there:
# jar tf $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar | grep TransactionSystemClient
org/apache/tephra/TransactionSystemClient.class
Do you have an idea why MR job cannot find this particular class?
If it helps, my table with secondary index is defined as follows:
0: jdbc:phoenix:localhost:2181/hbase> create table t1(v1 varchar, v2 varchar, v3 integer constraint primary_key primary key(v1)) immutable_rows=true, compression='SNAPPY';
1: jdbc:phoenix:localhost:2181/hbase> create index glb_idx on t1(v2) async;
And I run IndexTool with the command
${HBASE_HOME}/bin/hbase org.apache.phoenix.mapreduce.index.IndexTool -dt T1 -it GLB_IDX -op /tmp
When I create index synchronously and upsert some data into the table, index is populated correctly, so Phoenix secondary index configuration looks OK.
java cloudera-cdh phoenix
asked Jan 2 at 14:47
belobelo
587
I have Cloudera CDH 5.14.2 cluster with Apache Phoenix Parcel installed (APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3).
I have a table containing secondary index and I would like to populate this index using IndexTool provided with Apache Phoenix. But this is giving me the following error:
19/01/02 13:58:10 INFO mapreduce.Job: The url to track the job: http://mor-master-01.triviadata.local:8088/proxy/application_1546422102410_0020/
19/01/02 13:58:10 INFO mapreduce.Job: Running job: job_1546422102410_0020
19/01/02 13:58:18 INFO mapreduce.Job: Job job_1546422102410_0020 running in uber mode : false
19/01/02 13:58:18 INFO mapreduce.Job: map 0% reduce 0%
19/01/02 13:58:22 INFO mapreduce.Job: Task Id : attempt_1546422102410_0020_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.tephra.TransactionSystemClient
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.phoenix.transaction.TransactionFactory$Provider.<clinit>(TransactionFactory.java:27)
at org.apache.phoenix.query.QueryServicesOptions.<clinit>(QueryServicesOptions.java:270)
at org.apache.phoenix.query.QueryServicesImpl.<init>(QueryServicesImpl.java:36)
at org.apache.phoenix.jdbc.PhoenixDriver.getQueryServices(PhoenixDriver.java:197)
at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:235)
at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:113)
at org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:58)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:180)
at org.apache.phoenix.mapreduce.PhoenixInputFormat.createRecordReader(PhoenixInputFormat.java:76)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:521)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
When I check my HBASE_CLASSPATH with command ${HBASE_HOME}/bin/hbase classpath, I see it contains the following jars:
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/conf
/usr/java/latest/lib/tools.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/*.jar
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/hadoop/.//*
$HBASE_HOME/lib/hadoop-hdfs/./
$HBASE_HOME/hadoop-hdfs/lib/*
$HBASE_HOME/hadoop-hdfs/.//*
$HBASE_HOME/hadoop-yarn/lib/*
$HBASE_HOME/hadoop-yarn/.//*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*
/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*
/etc/hadoop/conf
$HBASE_HOME/lib/hadoop/*
$HBASE_HOME/lib/hadoop/lib/*
$HBASE_HOME/lib/zookeeper/*
$HBASE_HOME/lib/zookeeper/lib/*
$HBASE_HOME/jars/hbase-common-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-client-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/htrace-core-3.0.4.jar
$HBASE_HOME/jars/htrace-core-3.2.0-incubating.jar
$HBASE_HOME/jars/htrace-core4-4.0.1-incubating.jar
$HBASE_HOME/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/hbase-server-1.2.0-cdh5.14.2.jar
$HBASE_HOME/jars/metrics-core-2.2.0.jar
$HBASE_HOME/jars/metrics-core-3.1.2.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
$PHOENIX_HOME/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
$PHOENIX_HOME/lib/phoenix/lib/joda-time-1.6.jar
$PHOENIX_HOME/lib/phoenix/lib/antlr-runtime-3.5.2.jar
When I check the source code and it's dependencies I see that the missing class is part of the $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
When I grep the content of this JAR for the missing class I see that it is there:
# jar tf $PHOENIX_HOME/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar | grep TransactionSystemClient
org/apache/tephra/TransactionSystemClient.class
Do you have an idea why MR job cannot find this particular class?
If it helps, my table with secondary index is defined as follows:
0: jdbc:phoenix:localhost:2181/hbase> create table t1(v1 varchar, v2 varchar, v3 integer constraint primary_key primary key(v1)) immutable_rows=true, compression='SNAPPY';
1: jdbc:phoenix:localhost:2181/hbase> create index glb_idx on t1(v2) async;
And I run IndexTool with the command
${HBASE_HOME}/bin/hbase org.apache.phoenix.mapreduce.index.IndexTool -dt T1 -it GLB_IDX -op /tmp
When I create index synchronously and upsert some data into the table, index is populated correctly, so Phoenix secondary index configuration looks OK.
java cloudera-cdh phoenix
java cloudera-cdh phoenix
asked Jan 2 at 14:47
belobelo
587
asked Jan 2 at 14:47
belobelo
587
asked Jan 2 at 14:47
belobelo
587
asked Jan 2 at 14:47
belobelo
587
asked Jan 2 at 14:47
belobelo
587
587
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
0
This is a problem of the missing JARs on the classpath of the map-reduce job started by this command. By the trial and error I put together list of dependencies required required to run IndexTool on CDH 5.14.2.
cat classpath.txt
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase/bin/../conf
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-common-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-client-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.0.4.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.2.0-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core4-4.0.1-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-server-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-2.2.0.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-3.1.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/joda-time-1.6.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/antlr-runtime-3.5.2.jar
/usr/local/idx-tool-classpath/disruptor-3.3.6.jar
Then using Cloudera Manager I added all these JARs to mapreduce.application.classpath property in mapred-site.xml on each worker node.
Some of these dependencies are also required to submit the MR job, so I set HADDOP_CLASSPATH on the edge node where I'm running job from to contain all these JARs.
export HADDOP_CLASSPATH=$(paste -s -d: classpath.txt)
Then I could run the job with the following command
hadoop jar /opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar org.apache.phoenix.mapreduce.index.IndexTool -s SCHEMA_NAME -dt TEST_TABLE -it INDEX_TABLE -op /tmp
answered Jan 10 at 18:37
belobelo
587
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008368%2fapache-phoenix-indextool-failing-with-java-lang-classnotfoundexception-org-apac%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
This is a problem of the missing JARs on the classpath of the map-reduce job started by this command. By the trial and error I put together list of dependencies required required to run IndexTool on CDH 5.14.2.
cat classpath.txt
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase/bin/../conf
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-common-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-client-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.0.4.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.2.0-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core4-4.0.1-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-server-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-2.2.0.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-3.1.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/joda-time-1.6.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/antlr-runtime-3.5.2.jar
/usr/local/idx-tool-classpath/disruptor-3.3.6.jar
Then using Cloudera Manager I added all these JARs to mapreduce.application.classpath property in mapred-site.xml on each worker node.
Some of these dependencies are also required to submit the MR job, so I set HADDOP_CLASSPATH on the edge node where I'm running job from to contain all these JARs.
export HADDOP_CLASSPATH=$(paste -s -d: classpath.txt)
Then I could run the job with the following command
hadoop jar /opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar org.apache.phoenix.mapreduce.index.IndexTool -s SCHEMA_NAME -dt TEST_TABLE -it INDEX_TABLE -op /tmp
answered Jan 10 at 18:37
belobelo
587
add a comment |
0
This is a problem of the missing JARs on the classpath of the map-reduce job started by this command. By the trial and error I put together list of dependencies required required to run IndexTool on CDH 5.14.2.
cat classpath.txt
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase/bin/../conf
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-common-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-client-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.0.4.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.2.0-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core4-4.0.1-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-server-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-2.2.0.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-3.1.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/joda-time-1.6.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/antlr-runtime-3.5.2.jar
/usr/local/idx-tool-classpath/disruptor-3.3.6.jar
Then using Cloudera Manager I added all these JARs to mapreduce.application.classpath property in mapred-site.xml on each worker node.
Some of these dependencies are also required to submit the MR job, so I set HADDOP_CLASSPATH on the edge node where I'm running job from to contain all these JARs.
export HADDOP_CLASSPATH=$(paste -s -d: classpath.txt)
Then I could run the job with the following command
hadoop jar /opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar org.apache.phoenix.mapreduce.index.IndexTool -s SCHEMA_NAME -dt TEST_TABLE -it INDEX_TABLE -op /tmp
answered Jan 10 at 18:37
belobelo
587
add a comment |
0
0
0
This is a problem of the missing JARs on the classpath of the map-reduce job started by this command. By the trial and error I put together list of dependencies required required to run IndexTool on CDH 5.14.2.
cat classpath.txt
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase/bin/../conf
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-common-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-client-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.0.4.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.2.0-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core4-4.0.1-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-server-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-2.2.0.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-3.1.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/joda-time-1.6.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/antlr-runtime-3.5.2.jar
/usr/local/idx-tool-classpath/disruptor-3.3.6.jar
Then using Cloudera Manager I added all these JARs to mapreduce.application.classpath property in mapred-site.xml on each worker node.
Some of these dependencies are also required to submit the MR job, so I set HADDOP_CLASSPATH on the edge node where I'm running job from to contain all these JARs.
export HADDOP_CLASSPATH=$(paste -s -d: classpath.txt)
Then I could run the job with the following command
hadoop jar /opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar org.apache.phoenix.mapreduce.index.IndexTool -s SCHEMA_NAME -dt TEST_TABLE -it INDEX_TABLE -op /tmp
answered Jan 10 at 18:37
belobelo
587
This is a problem of the missing JARs on the classpath of the map-reduce job started by this command. By the trial and error I put together list of dependencies required required to run IndexTool on CDH 5.14.2.
cat classpath.txt
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hbase/bin/../conf
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-common-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-client-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-hadoop-compat-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.0.4.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core-3.2.0-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/htrace-core4-4.0.1-incubating.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-protocol-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/hbase-server-1.2.0-cdh5.14.2.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-2.2.0.jar
/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/metrics-core-3.1.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-hbase-compat-1.2-cdh-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-api-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/tephra-core-0.14.0-incubating.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-zookeeper-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-api-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/twill-discovery-core-0.8.0.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/joda-time-1.6.jar
/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/antlr-runtime-3.5.2.jar
/usr/local/idx-tool-classpath/disruptor-3.3.6.jar
Then using Cloudera Manager I added all these JARs to mapreduce.application.classpath property in mapred-site.xml on each worker node.
Some of these dependencies are also required to submit the MR job, so I set HADDOP_CLASSPATH on the edge node where I'm running job from to contain all these JARs.
export HADDOP_CLASSPATH=$(paste -s -d: classpath.txt)
Then I could run the job with the following command
hadoop jar /opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.14.2.p0.3/lib/phoenix/lib/phoenix-core-4.14.0-cdh5.14.2.jar org.apache.phoenix.mapreduce.index.IndexTool -s SCHEMA_NAME -dt TEST_TABLE -it INDEX_TABLE -op /tmp
answered Jan 10 at 18:37
belobelo
587
answered Jan 10 at 18:37
belobelo
587
answered Jan 10 at 18:37
belobelo
587
answered Jan 10 at 18:37
belobelo
587
587
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008368%2fapache-phoenix-indextool-failing-with-java-lang-classnotfoundexception-org-apac%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
