Parsing incoming Json message coming from a NiFi Stream into Hbase table

Multi tool use
Multi tool use












0















​Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):



{   "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }


As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.



What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.



enter image description here










share|improve this question




















  • 1





    Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)

    – Harold
    Dec 31 '18 at 9:04
















0















​Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):



{   "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }


As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.



What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.



enter image description here










share|improve this question




















  • 1





    Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)

    – Harold
    Dec 31 '18 at 9:04














0












0








0








​Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):



{   "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }


As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.



What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.



enter image description here










share|improve this question
















​Hi All,
I am getting a Stream of message from a Kafka Topic in NiFi which I am reading via a consumer process. Message is in the format of json (dummy json values, json format same as original one):



{   "schema": {
"type": "struct",
"name": "emp_table",
"fields": [
{
"field": "emp_id",
"type": "string"
},
{
"field": "emp_name",
"type": "String"
},
{
"field": "city",
"type": "string"
},
{
"field": "emp_sal",
"type": "string"
},
{
"field": "manager_name",
"type": "string"
}
] }, "payload": {
"emp_id": "1",
"emp_name": "abc",
"city": "NYK",
"emp_sal": "100000",
"manager_name": "xyz" } }


As u can see here actual table name is under Schema and column values is under payload. I am able to parse column values and put into Hbase table by using EvaluateJsonPath and PutHBaseJson processors in NiFi.



What I am able to achieve is manually putting table name and rowid. But my issue is I want to fetch tablename(in above example emp_table) and a rowid (in above example emp_id) from the json and at runtime provide those values to PutHbaseJson processor in NiFi along with the column values.



enter image description here







json parsing hbase apache-nifi






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 2 at 2:20









Dale Burrell

3,10032351




3,10032351










asked Dec 29 '18 at 21:12









Appy22Appy22

63




63








  • 1





    Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)

    – Harold
    Dec 31 '18 at 9:04














  • 1





    Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)

    – Harold
    Dec 31 '18 at 9:04








1




1





Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)

– Harold
Dec 31 '18 at 9:04





Instead of using PutHBaseJson processor, you can try using InvokeHTTP processor with HBase REST. Advantage is that you can probably build a dynamic URL with tablename and row id. Inconvenient is the input data format encoded in byte64 (unless you decide to store a JSON object in a single column)

– Harold
Dec 31 '18 at 9:04












1 Answer
1






active

oldest

votes


















0














You should be able to add another JSON path expression to EvaluateJsonPath with something like:



table = $.schema.name


Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.






share|improve this answer























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53973373%2fparsing-incoming-json-message-coming-from-a-nifi-stream-into-hbase-table%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    You should be able to add another JSON path expression to EvaluateJsonPath with something like:



    table = $.schema.name


    Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.






    share|improve this answer




























      0














      You should be able to add another JSON path expression to EvaluateJsonPath with something like:



      table = $.schema.name


      Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.






      share|improve this answer


























        0












        0








        0







        You should be able to add another JSON path expression to EvaluateJsonPath with something like:



        table = $.schema.name


        Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.






        share|improve this answer













        You should be able to add another JSON path expression to EvaluateJsonPath with something like:



        table = $.schema.name


        Then in PutHBaseJson make the table name ${table}, or whatever you named it in EvaluateJsonPath.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 2 at 1:25









        Bryan BendeBryan Bende

        10.8k917




        10.8k917






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53973373%2fparsing-incoming-json-message-coming-from-a-nifi-stream-into-hbase-table%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            ygYIr02TSm4EYQEPxn6f,Tk,Pryc P,F2kQGFy8SUol8vEl
            m8I8f,MIGg

            Popular posts from this blog

            Monofisismo

            Angular Downloading a file using contenturl with Basic Authentication

            Olmecas