Parsing incoming Json message coming from a NiFi Stream into Hbase table
![Multi tool use Multi tool use](http://sgv.ssvwv.com/sg/ssvwvcomimagb.png)
Multi tool use
Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):
{ "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }
As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.
What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.
json parsing hbase apache-nifi
add a comment |
Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):
{ "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }
As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.
What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.
json parsing hbase apache-nifi
1
Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)
– Harold
Dec 31 '18 at 9:04
add a comment |
Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):
{ "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }
As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.
What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.
json parsing hbase apache-nifi
Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):
{ "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }
As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.
What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.
json parsing hbase apache-nifi
json parsing hbase apache-nifi
edited Jan 2 at 2:20
Dale Burrell
3,10032351
3,10032351
asked Dec 29 '18 at 21:12
Appy22Appy22
63
63
1
Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)
– Harold
Dec 31 '18 at 9:04
add a comment |
1
Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)
– Harold
Dec 31 '18 at 9:04
1
1
Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)
– Harold
Dec 31 '18 at 9:04
Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)
– Harold
Dec 31 '18 at 9:04
add a comment |
1 Answer
1
active
oldest
votes
You should be able to add another JSON path expression to EvaluateJsonPath with something like:
table = $.schema.name
Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53973373%2fparsing-incoming-json-message-coming-from-a-nifi-stream-into-hbase-table%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You should be able to add another JSON path expression to EvaluateJsonPath with something like:
table = $.schema.name
Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.
add a comment |
You should be able to add another JSON path expression to EvaluateJsonPath with something like:
table = $.schema.name
Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.
add a comment |
You should be able to add another JSON path expression to EvaluateJsonPath with something like:
table = $.schema.name
Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.
You should be able to add another JSON path expression to EvaluateJsonPath with something like:
table = $.schema.name
Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.
answered Jan 2 at 1:25
![](https://i.stack.imgur.com/X8GdE.jpg?s=32&g=1)
![](https://i.stack.imgur.com/X8GdE.jpg?s=32&g=1)
Bryan BendeBryan Bende
10.8k917
10.8k917
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53973373%2fparsing-incoming-json-message-coming-from-a-nifi-stream-into-hbase-table%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
ygYIr02TSm4EYQEPxn6f,Tk,Pryc P,F2kQGFy8SUol8vEl
1
Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)
– Harold
Dec 31 '18 at 9:04