How To Harden And Simplify Urlopen Function In Python

March 22nd 2021

5

Python has a useful high-level HTTP client built into the standard library:

urllib.request.urlopen()

urlopen

0 reactions

However, there is a notable security concern with

urlopen

file:///

urlopen 

https

0 reactions

The built-in OpenerDirector class offers just such an opportunity to streamline

urlopen

0 reactions

Security problems with the default

urlopen

You might try the following on a Linux system:

0 reactions

from urllib.request import urlopen

url = "file:///etc/passwd"

with urlopen(url) as response:
  print(response.read().decode())

A little disturbing, yes? Bandit agrees. (Perhaps you want to consider scanning with Bandit or its related flake8 plugin.)

0 reactions

The code above certainly communicates the lesson "sanitize your user inputs". Of course, if you control that "url" string, or can ensure that it starts with the correct "https://" scheme, then things are looking better. If this url comes from user input, though, it would be good to check for protocol at least, and, better yet, ensure that the domain is as expected.

0 reactions

A suggested solution

The following code results in a

urlopen 

0 reactions

import urllib.request


class SafeOpener(urllib.request.OpenerDirector):
    def __init__(self, handlers: typing.Iterable = None):
        super().__init__()
        handlers = handlers or (
            urllib.request.UnknownHandler,
            urllib.request.HTTPDefaultErrorHandler,
            urllib.request.HTTPRedirectHandler,
            urllib.request.HTTPSHandler,
            urllib.request.HTTPErrorProcessor,
        )

        for handler_class in handlers:
            self.add_handler(handler_class())


opener = SafeOpener()
urllib.request.install_opener(opener)

After running the above, using

urllib.request.urlopen()

URLError

urlopen

0 reactions

By the way, if you prefer not to override the opener in the

urlopen 

install_opener(opener)

opener.open()

urlopen()

0 reactions

The above code assumes that all HTTP calls will be encrypted with TLS (aka "SSL", with "https:" at the beginning of the URL). That also means that testing will need to use "https:" URLs as well. Consider using the vcr library to mock-reproduce HTTP calls, or the trustme library to actually set up certs for testing. If you need to use unencrypted "http:" URLs, though, you can simply add

urllib.request.HTTPHandler

handlers 

0 reactions

The handlers, defined

This is the chain of default handlers normally used by

urlopen

0 reactions

• ProxyHandler

  : searches system settings for proxies. If you are 100% sure that your tool or library will never be used with a proxy, then this is not necessary.

• UnknownHandler

  : raises a URLError if the protocol requested in the URL is not supported by a handler in this chain. Very helpful and recommended.

• HTTPHandler

  : handles unencrypted HTTP connections. Only add this if you are sure you need URL support other than HTTPS

• HTTPDefaultErrorHandler

  : a prefilter of sorts that turns all responses into exceptions, for handling downstream. This is necessary unless you plan on handling statuses, exceptions, and redirects yourself.

• HTTPRedirectHandler

  : handles redirects (status codes 301, 302, 303, or 307) and is necessary if automatic following of redirects is desired.

• FTPHandler

  : handles ftp: URLs. Not necessary for HTTP calls.

• FileHandler

  : handles file: URLs, and poses security risks. Rarely should this be necessary, if ever.

• HTTPErrorProcessor

  : The final response handler, raising any non-200 (OK) responses

• DataHandler

  : handles 

  data:

   URLs. Hard to imagine why this would be necessary in normal use, and could pose potential security risks with user input.

As may be apparent, several of the above are rarely necessary, if ever, for HTTP API work. Instead, I recommend this list as a happy medium between security and usability:

0 reactions

• ProxyHandler

• UnknownHandler

• HTTPHandler

• HTTPDefaultErrorHandler

• HTTPRedirectHandler

• HTTPSHandler

• HTTPErrorProcessor

Of the above,

ProxyHandler 

HTTPHandler 

0 reactions

• UnknownHandler

• HTTPDefaultErrorHandler

• HTTPRedirectHandler

• HTTPSHandler

• HTTPErrorProcessor

Five handlers, not the original nine.

0 reactions

Flexibility

The use cases for custom

OpenerDirector 

BaseHandler

http_error_401

0 reactions

I hope these suggestions open up possibilities and peace of mind for you.

0 reactions

Also published at https://dev.to/bowmanjd/hardening-and-simplifying-python-s-urlopen-4gee

0 reactions

5