org.apache.avro.SchemaParseException: Undefined name
My spark job is reading from kafka and decoding the message. my spark job is running well on my local spark cluster (https://archive.apache.org/dist/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz) on MAC. but when run the job against spark (2.2.1-hadoop-2.7, from https://hub.docker.com/r/gettyimages/spark/tags) cluster on k8s and openstack, I got the following error:
Caused by: org.apache.avro.SchemaParseException: Undefined name: "my_items"
at org.apache.avro.Schema.parse(Schema.java:1052)
at org.apache.avro.Schema.parse(Schema.java:1164)
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema$Parser.parse(Schema.java:927)
at org.apache.avro.Schema$Parser.parse(Schema.java:917)
I am pretty sure that in two cases, spark job is trying to parse the same schema. why spark job failed on my k8s spark cluster? I am really confused and have no idea how to debug for now. any hints welcomed. Thanks
scala apache-spark
kubernetes avro openstack-nova asked Jan 3 at 19:35
BAEBAE
2,92563176
add a comment |
My spark job is reading from kafka and decoding the message. my spark job is running well on my local spark cluster (https://archive.apache.org/dist/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz) on MAC. but when run the job against spark (2.2.1-hadoop-2.7, from https://hub.docker.com/r/gettyimages/spark/tags) cluster on k8s and openstack, I got the following error:
Caused by: org.apache.avro.SchemaParseException: Undefined name: "my_items"
at org.apache.avro.Schema.parse(Schema.java:1052)
at org.apache.avro.Schema.parse(Schema.java:1164)
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema$Parser.parse(Schema.java:927)
at org.apache.avro.Schema$Parser.parse(Schema.java:917)
I am pretty sure that in two cases, spark job is trying to parse the same schema. why spark job failed on my k8s spark cluster? I am really confused and have no idea how to debug for now. any hints welcomed. Thanks
scala apache-spark
kubernetes avro openstack-nova asked Jan 3 at 19:35
BAEBAE
2,92563176
add a comment |
My spark job is reading from kafka and decoding the message. my spark job is running well on my local spark cluster (https://archive.apache.org/dist/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz) on MAC. but when run the job against spark (2.2.1-hadoop-2.7, from https://hub.docker.com/r/gettyimages/spark/tags) cluster on k8s and openstack, I got the following error:
Caused by: org.apache.avro.SchemaParseException: Undefined name: "my_items"
at org.apache.avro.Schema.parse(Schema.java:1052)
at org.apache.avro.Schema.parse(Schema.java:1164)
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema$Parser.parse(Schema.java:927)
at org.apache.avro.Schema$Parser.parse(Schema.java:917)
I am pretty sure that in two cases, spark job is trying to parse the same schema. why spark job failed on my k8s spark cluster? I am really confused and have no idea how to debug for now. any hints welcomed. Thanks
scala apache-spark
kubernetes avro openstack-nova asked Jan 3 at 19:35
BAEBAE
2,92563176
My spark job is reading from kafka and decoding the message. my spark job is running well on my local spark cluster (https://archive.apache.org/dist/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz) on MAC. but when run the job against spark (2.2.1-hadoop-2.7, from https://hub.docker.com/r/gettyimages/spark/tags) cluster on k8s and openstack, I got the following error:
Caused by: org.apache.avro.SchemaParseException: Undefined name: "my_items"
at org.apache.avro.Schema.parse(Schema.java:1052)
at org.apache.avro.Schema.parse(Schema.java:1164)
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema$Parser.parse(Schema.java:927)
at org.apache.avro.Schema$Parser.parse(Schema.java:917)
I am pretty sure that in two cases, spark job is trying to parse the same schema. why spark job failed on my k8s spark cluster? I am really confused and have no idea how to debug for now. any hints welcomed. Thanks
scala apache-spark
kubernetes avro openstack-nova scala apache-spark
kubernetes avro openstack-nova asked Jan 3 at 19:35
BAEBAE
2,92563176
asked Jan 3 at 19:35
BAEBAE
2,92563176
asked Jan 3 at 19:35
BAEBAE
2,92563176
asked Jan 3 at 19:35
BAEBAE
2,92563176
asked Jan 3 at 19:35
BAEBAE
2,92563176
2,92563176
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
It's likely different dependencies for Avro. 1.7.4 vs 1.7.7.
The container image:
$ find / | grep avro
/usr/spark-2.2.1/examples/src/main/python/avro_inputformat.py
/usr/spark-2.2.1/examples/src/main/resources/users.avro
/usr/spark-2.2.1/jars/avro-mapred-1.7.7-hadoop2.jar
/usr/hadoop-2.7.3/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/tools/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar
The spark-2.2.1-bin-hadoop2.7.tgz file:
$ find . | grep avro
./spark-2.2.1-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py
./spark-2.2.1-bin-hadoop2.7/examples/src/main/resources/users.avro
./spark-2.2.1-bin-hadoop2.7/jars/avro-mapred-1.7.7-hadoop2.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-1.7.7.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-ipc-1.7.7.jar
# Hadoop Avro?
answered Jan 3 at 22:16
RicoRico
29.3k95471
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54028707%2forg-apache-avro-schemaparseexception-undefined-name%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
It's likely different dependencies for Avro. 1.7.4 vs 1.7.7.
The container image:
$ find / | grep avro
/usr/spark-2.2.1/examples/src/main/python/avro_inputformat.py
/usr/spark-2.2.1/examples/src/main/resources/users.avro
/usr/spark-2.2.1/jars/avro-mapred-1.7.7-hadoop2.jar
/usr/hadoop-2.7.3/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/tools/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar
The spark-2.2.1-bin-hadoop2.7.tgz file:
$ find . | grep avro
./spark-2.2.1-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py
./spark-2.2.1-bin-hadoop2.7/examples/src/main/resources/users.avro
./spark-2.2.1-bin-hadoop2.7/jars/avro-mapred-1.7.7-hadoop2.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-1.7.7.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-ipc-1.7.7.jar
# Hadoop Avro?
answered Jan 3 at 22:16
RicoRico
29.3k95471
add a comment |
It's likely different dependencies for Avro. 1.7.4 vs 1.7.7.
The container image:
$ find / | grep avro
/usr/spark-2.2.1/examples/src/main/python/avro_inputformat.py
/usr/spark-2.2.1/examples/src/main/resources/users.avro
/usr/spark-2.2.1/jars/avro-mapred-1.7.7-hadoop2.jar
/usr/hadoop-2.7.3/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/tools/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar
The spark-2.2.1-bin-hadoop2.7.tgz file:
$ find . | grep avro
./spark-2.2.1-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py
./spark-2.2.1-bin-hadoop2.7/examples/src/main/resources/users.avro
./spark-2.2.1-bin-hadoop2.7/jars/avro-mapred-1.7.7-hadoop2.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-1.7.7.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-ipc-1.7.7.jar
# Hadoop Avro?
answered Jan 3 at 22:16
RicoRico
29.3k95471
add a comment |
It's likely different dependencies for Avro. 1.7.4 vs 1.7.7.
The container image:
$ find / | grep avro
/usr/spark-2.2.1/examples/src/main/python/avro_inputformat.py
/usr/spark-2.2.1/examples/src/main/resources/users.avro
/usr/spark-2.2.1/jars/avro-mapred-1.7.7-hadoop2.jar
/usr/hadoop-2.7.3/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/tools/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar
The spark-2.2.1-bin-hadoop2.7.tgz file:
$ find . | grep avro
./spark-2.2.1-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py
./spark-2.2.1-bin-hadoop2.7/examples/src/main/resources/users.avro
./spark-2.2.1-bin-hadoop2.7/jars/avro-mapred-1.7.7-hadoop2.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-1.7.7.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-ipc-1.7.7.jar
# Hadoop Avro?
answered Jan 3 at 22:16
RicoRico
29.3k95471
It's likely different dependencies for Avro. 1.7.4 vs 1.7.7.
The container image:
$ find / | grep avro
/usr/spark-2.2.1/examples/src/main/python/avro_inputformat.py
/usr/spark-2.2.1/examples/src/main/resources/users.avro
/usr/spark-2.2.1/jars/avro-mapred-1.7.7-hadoop2.jar
/usr/hadoop-2.7.3/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/common/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/tools/lib/avro-1.7.4.jar
/usr/hadoop-2.7.3/share/hadoop/mapreduce/lib/avro-1.7.4.jar
The spark-2.2.1-bin-hadoop2.7.tgz file:
$ find . | grep avro
./spark-2.2.1-bin-hadoop2.7/examples/src/main/python/avro_inputformat.py
./spark-2.2.1-bin-hadoop2.7/examples/src/main/resources/users.avro
./spark-2.2.1-bin-hadoop2.7/jars/avro-mapred-1.7.7-hadoop2.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-1.7.7.jar
./spark-2.2.1-bin-hadoop2.7/jars/avro-ipc-1.7.7.jar
# Hadoop Avro?
answered Jan 3 at 22:16
RicoRico
29.3k95471
answered Jan 3 at 22:16
RicoRico
29.3k95471
answered Jan 3 at 22:16
RicoRico
29.3k95471
answered Jan 3 at 22:16
RicoRico
29.3k95471
29.3k95471
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54028707%2forg-apache-avro-schemaparseexception-undefined-name%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
