How do I (safely) send a Python object to my Flask API?

Multi tool use
I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.
I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier
object from sklearn.ensemble
, but this could be any of a wide variety of object types.
So far I have tried to json.dumps()
my object, but this object is not JSON serializable. I have also tried to pickle.dumps()
my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads()
on a possibly malicious object.
Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?
The script performing the POST request:
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model)
r = requests.post(url,data=data)
The Flask API:
@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():
model = pickle.loads(request.get_data())
This setup actually causes an error when trying to decode the data with pickle:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3
Is there a nice and safe way to do is?
python json scikit-learn pickle
add a comment |
I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.
I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier
object from sklearn.ensemble
, but this could be any of a wide variety of object types.
So far I have tried to json.dumps()
my object, but this object is not JSON serializable. I have also tried to pickle.dumps()
my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads()
on a possibly malicious object.
Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?
The script performing the POST request:
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model)
r = requests.post(url,data=data)
The Flask API:
@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():
model = pickle.loads(request.get_data())
This setup actually causes an error when trying to decode the data with pickle:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3
Is there a nice and safe way to do is?
python json scikit-learn pickle
don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…
– Urban48
Jan 2 at 16:12
Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.
– Lars S
Jan 2 at 16:17
in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start
– Urban48
Jan 2 at 16:21
As you can see I have tried that with theRandomForestClassifier
object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).
– Lars S
Jan 2 at 16:24
@Urban48 but not all objects are JSON serializable.
– juanpa.arrivillaga
Jan 2 at 16:47
add a comment |
I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.
I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier
object from sklearn.ensemble
, but this could be any of a wide variety of object types.
So far I have tried to json.dumps()
my object, but this object is not JSON serializable. I have also tried to pickle.dumps()
my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads()
on a possibly malicious object.
Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?
The script performing the POST request:
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model)
r = requests.post(url,data=data)
The Flask API:
@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():
model = pickle.loads(request.get_data())
This setup actually causes an error when trying to decode the data with pickle:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3
Is there a nice and safe way to do is?
python json scikit-learn pickle
I am currently trying to build a Flask Web API that is able to receive a python object in a POST-request.
I am using Python 3.7.1 for creating the request and Python 2.7 for running the API. The API is set up to run on my local machine. The object I am trying to send to my API is a RandomForestClassifier
object from sklearn.ensemble
, but this could be any of a wide variety of object types.
So far I have tried to json.dumps()
my object, but this object is not JSON serializable. I have also tried to pickle.dumps()
my object, but this caused an error when trying to load the object on the API side. Also, since the API will receive requests from anonymous users, I am worried about performing a pickle.loads()
on a possibly malicious object.
Is this a grounded worry? And if so, what is the best way to send any python object in a POST request?
The script performing the POST request:
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model)
r = requests.post(url,data=data)
The Flask API:
@app.route('/flask-api-function', methods=['POST'])
def flask_api_function():
model = pickle.loads(request.get_data())
This setup actually causes an error when trying to decode the data with pickle:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/flask/app.py", line 2309, in __call__
return self.wsgi_app(environ, start_response)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2295, in wsgi_app
response = self.handle_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1741, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Library/Python/2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/Library/Python/2.7/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/larssuanet/Documents/enjins/dscs/flask_api.py", line 39, in store_model
model = pickle.loads(request.get_data())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 886, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3
Is there a nice and safe way to do is?
python json scikit-learn pickle
python json scikit-learn pickle
edited Jan 2 at 16:44
davidism
65.2k12175189
65.2k12175189
asked Jan 2 at 16:01
Lars SLars S
7310
7310
don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…
– Urban48
Jan 2 at 16:12
Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.
– Lars S
Jan 2 at 16:17
in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start
– Urban48
Jan 2 at 16:21
As you can see I have tried that with theRandomForestClassifier
object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).
– Lars S
Jan 2 at 16:24
@Urban48 but not all objects are JSON serializable.
– juanpa.arrivillaga
Jan 2 at 16:47
add a comment |
don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…
– Urban48
Jan 2 at 16:12
Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.
– Lars S
Jan 2 at 16:17
in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start
– Urban48
Jan 2 at 16:21
As you can see I have tried that with theRandomForestClassifier
object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).
– Lars S
Jan 2 at 16:24
@Urban48 but not all objects are JSON serializable.
– juanpa.arrivillaga
Jan 2 at 16:47
don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…
– Urban48
Jan 2 at 16:12
don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…
– Urban48
Jan 2 at 16:12
Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.
– Lars S
Jan 2 at 16:17
Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.
– Lars S
Jan 2 at 16:17
in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start
– Urban48
Jan 2 at 16:21
in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start
– Urban48
Jan 2 at 16:21
As you can see I have tried that with the
RandomForestClassifier
object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).– Lars S
Jan 2 at 16:24
As you can see I have tried that with the
RandomForestClassifier
object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).– Lars S
Jan 2 at 16:24
@Urban48 but not all objects are JSON serializable.
– juanpa.arrivillaga
Jan 2 at 16:47
@Urban48 but not all objects are JSON serializable.
– juanpa.arrivillaga
Jan 2 at 16:47
add a comment |
1 Answer
1
active
oldest
votes
When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)
r = requests.post(url,data=data)
Generally I would try to find a way to serialize the object into JSON since pickle has major security risks
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009432%2fhow-do-i-safely-send-a-python-object-to-my-flask-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)
r = requests.post(url,data=data)
Generally I would try to find a way to serialize the object into JSON since pickle has major security risks
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
add a comment |
When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)
r = requests.post(url,data=data)
Generally I would try to find a way to serialize the object into JSON since pickle has major security risks
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
add a comment |
When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)
r = requests.post(url,data=data)
Generally I would try to find a way to serialize the object into JSON since pickle has major security risks
When you pickle the object in python 3, pass the protocol keyword and set it to 2. This will ensure that it will work on python 2.
import requests
import pickle
url = "http://localhost:5000/flask-api-function"
# the object I want to send is the 'model' object
data = pickle.dumps(model,protocol=2)
r = requests.post(url,data=data)
Generally I would try to find a way to serialize the object into JSON since pickle has major security risks
answered Jan 2 at 16:45
Dani GDani G
517412
517412
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
add a comment |
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
Thanks, this will solve my error for now. But like you said, I will still be looking out for a more secure way to do this.
– Lars S
Jan 3 at 8:07
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009432%2fhow-do-i-safely-send-a-python-object-to-my-flask-api%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
NigMtXaLNUuQtkz3vVA Gx,I CCZW,ePTWfEDcZhbGWh5Vv5V,tdglx,7y9jsOs K
don't send the actual object, its bad practice, and probably will get you in a never ending maintenance fire fighting. see if you can send a representation of them, and reconstruct them on the other side.. also see: stackoverflow.com/questions/13031058/…
– Urban48
Jan 2 at 16:12
Any suggestions on how someone could go about doing that when they want to support a wide variety of objects? I feel like there should be an easier way to do this without having to write logic for reconstructing every single object-type you want to support.
– Lars S
Jan 2 at 16:17
in python everything is an object... can you be more specific? try to serialize your objects into JSON, thats a good place to start
– Urban48
Jan 2 at 16:21
As you can see I have tried that with the
RandomForestClassifier
object, and this object is not serializable. I don't need a solution for this specific object though, I am looking for a clean way to serialize a wide variety of objects (specifically ML models).– Lars S
Jan 2 at 16:24
@Urban48 but not all objects are JSON serializable.
– juanpa.arrivillaga
Jan 2 at 16:47