How do I (safely) send a Python object to my Flask API?

Multi tool use
Multi tool use












0















I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.



I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier object from sklearn.ensemble, but this could be any of a wide variety of object types.



So far I have tried to json.dumps() my object, but this object is not JSON serializable. I have also tried to pickle.dumps() my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads() on a possibly malicious object.



Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?



The script performing the POST request:



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model)

r = requests.post(url,data=data)


The Flask API:



@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():

model = pickle.loads(request.get_data())


This setup actually causes an error when trying to decode the data with pickle:



Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3


Is there a nice and safe way to do is?










share|improve this question

























  • don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…

    – Urban48
    Jan 2 at 16:12













  • Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.

    – Lars S
    Jan 2 at 16:17











  • in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start

    – Urban48
    Jan 2 at 16:21











  • As you can see I have tried that with the RandomForestClassifier object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).

    – Lars S
    Jan 2 at 16:24











  • @Urban48 but not all objects are JSON serializable.

    – juanpa.arrivillaga
    Jan 2 at 16:47
















0















I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.



I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier object from sklearn.ensemble, but this could be any of a wide variety of object types.



So far I have tried to json.dumps() my object, but this object is not JSON serializable. I have also tried to pickle.dumps() my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads() on a possibly malicious object.



Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?



The script performing the POST request:



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model)

r = requests.post(url,data=data)


The Flask API:



@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():

model = pickle.loads(request.get_data())


This setup actually causes an error when trying to decode the data with pickle:



Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3


Is there a nice and safe way to do is?










share|improve this question

























  • don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…

    – Urban48
    Jan 2 at 16:12













  • Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.

    – Lars S
    Jan 2 at 16:17











  • in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start

    – Urban48
    Jan 2 at 16:21











  • As you can see I have tried that with the RandomForestClassifier object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).

    – Lars S
    Jan 2 at 16:24











  • @Urban48 but not all objects are JSON serializable.

    – juanpa.arrivillaga
    Jan 2 at 16:47














0












0








0








I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.



I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier object from sklearn.ensemble, but this could be any of a wide variety of object types.



So far I have tried to json.dumps() my object, but this object is not JSON serializable. I have also tried to pickle.dumps() my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads() on a possibly malicious object.



Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?



The script performing the POST request:



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model)

r = requests.post(url,data=data)


The Flask API:



@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():

model = pickle.loads(request.get_data())


This setup actually causes an error when trying to decode the data with pickle:



Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3


Is there a nice and safe way to do is?










share|improve this question
















I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.



I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier object from sklearn.ensemble, but this could be any of a wide variety of object types.



So far I have tried to json.dumps() my object, but this object is not JSON serializable. I have also tried to pickle.dumps() my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads() on a possibly malicious object.



Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?



The script performing the POST request:



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model)

r = requests.post(url,data=data)


The Flask API:



@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():

model = pickle.loads(request.get_data())


This setup actually causes an error when trying to decode the data with pickle:



Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3


Is there a nice and safe way to do is?







python json scikit-learn pickle






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 2 at 16:44









davidism

65.2k12175189




65.2k12175189










asked Jan 2 at 16:01









Lars SLars S

7310




7310













  • don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…

    – Urban48
    Jan 2 at 16:12













  • Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.

    – Lars S
    Jan 2 at 16:17











  • in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start

    – Urban48
    Jan 2 at 16:21











  • As you can see I have tried that with the RandomForestClassifier object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).

    – Lars S
    Jan 2 at 16:24











  • @Urban48 but not all objects are JSON serializable.

    – juanpa.arrivillaga
    Jan 2 at 16:47



















  • don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…

    – Urban48
    Jan 2 at 16:12













  • Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.

    – Lars S
    Jan 2 at 16:17











  • in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start

    – Urban48
    Jan 2 at 16:21











  • As you can see I have tried that with the RandomForestClassifier object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).

    – Lars S
    Jan 2 at 16:24











  • @Urban48 but not all objects are JSON serializable.

    – juanpa.arrivillaga
    Jan 2 at 16:47

















don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…

– Urban48
Jan 2 at 16:12







don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…

– Urban48
Jan 2 at 16:12















Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.

– Lars S
Jan 2 at 16:17





Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.

– Lars S
Jan 2 at 16:17













in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start

– Urban48
Jan 2 at 16:21





in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start

– Urban48
Jan 2 at 16:21













As you can see I have tried that with the RandomForestClassifier object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).

– Lars S
Jan 2 at 16:24





As you can see I have tried that with the RandomForestClassifier object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).

– Lars S
Jan 2 at 16:24













@Urban48 but not all objects are JSON serializable.

– juanpa.arrivillaga
Jan 2 at 16:47





@Urban48 but not all objects are JSON serializable.

– juanpa.arrivillaga
Jan 2 at 16:47












1 Answer
1






active

oldest

votes


















1














When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)

r = requests.post(url,data=data)


Generally I would try to find a way to serialize the object into JSON since pickle has major security risks






share|improve this answer
























  • Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

    – Lars S
    Jan 3 at 8:07











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009432%2fhow-do-i-safely-send-a-python-object-to-my-flask-api%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)

r = requests.post(url,data=data)


Generally I would try to find a way to serialize the object into JSON since pickle has major security risks






share|improve this answer
























  • Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

    – Lars S
    Jan 3 at 8:07
















1














When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)

r = requests.post(url,data=data)


Generally I would try to find a way to serialize the object into JSON since pickle has major security risks






share|improve this answer
























  • Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

    – Lars S
    Jan 3 at 8:07














1












1








1







When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)

r = requests.post(url,data=data)


Generally I would try to find a way to serialize the object into JSON since pickle has major security risks






share|improve this answer













When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.



import requests
import pickle

url = "http://localhost:5000/flask-api-function"

# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)

r = requests.post(url,data=data)


Generally I would try to find a way to serialize the object into JSON since pickle has major security risks







share|improve this answer












share|improve this answer



share|improve this answer










answered Jan 2 at 16:45









Dani GDani G

517412




517412













  • Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

    – Lars S
    Jan 3 at 8:07



















  • Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

    – Lars S
    Jan 3 at 8:07

















Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

– Lars S
Jan 3 at 8:07





Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.

– Lars S
Jan 3 at 8:07




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009432%2fhow-do-i-safely-send-a-python-object-to-my-flask-api%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







NigMtXaLNUuQtkz3vVA Gx,I CCZW,ePTWfEDcZhbGWh5Vv5V,tdglx,7y9jsOs K
Ymx6 4f7ENeRSrfi,7gV,95w6041A7iWTq

Popular posts from this blog

Monofisismo

Angular Downloading a file using contenturl with Basic Authentication

Olmecas