>>> import requests >>> requests.get(user_supplied_url, timeout=10) What's the maximum amount of time this could take to complete? 7 atbildes
 |
Pēteris Caune (2022-08-16 14:04:31) |
>>> import requests >>> requests.get(user_supplied_url, timeout=10) What's the maximum amount of time this could take to complete? | |
|
Pēteris Caune (2022-08-16 14:21:33) |
"10 seconds or thereabouts" is of course wrong :-) The "timeout" parameter specifies the max time gap between socket events, *not* the time limit for completing the entire request. | |
|
Pēteris Caune (2022-08-16 14:22:13) |
A naughty remote server can send the response byte-by-byte, with a second of pause between each sent byte, and force the request to take an arbitrarily long time. Example: https://t.co/TNVu5zAbLO | |
|
Pēteris Caune (2022-08-16 14:37:33) |
So how do you implement a time limit for the entire request? The linked SO answer suggests using sys.settrace(), measuring elapsed time in the trace function, and throwing an exception when over time budget. This can be a context manager even: https://t.co/vF1BpOL0Qf | |
|
Pēteris Caune (2022-08-16 14:55:16) |
This seems to work, but I'm not comfortable with injecting phantom exceptions at random points in requests' execution flow. This could potentially cause requests to crash in new and unpredictable ways. | |
|
Pēteris Caune (2022-08-16 14:55:59) |
Other options: * run the request in a separate process (using multiprocessing). Terminate the process when over time limit * use aiohttp instead * use pycurl instead | |
|
Pēteris Caune (2022-08-16 15:08:19) |
Seeing how mature and versatile curl and libcurl is, I'm exploring the pycurl option. I wrote a pycurl wrapper which mimics the requests interface (not all of it, just the parts I need) https://t.co/iFoU4xH5hK | |
|
Pēteris Caune (2022-08-16 15:12:36) |
Target URLs, headers, POST data is user controlled and untrusted, so some security precautions: * only allow http and https protocols * refuse requests to private IP ranges * limit the number of redirects * timeout! | |