GitHub - constverum/ProxyBroker: Proxy [Finder | Checker | Server]. HTTP(S) &...
source link: https://github.com/constverum/ProxyBroker
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
README.rst
ProxyBroker
ProxyBroker is an open source tool that asynchronously finds public proxies from multiple sources and concurrently checks them.
Features
- Finds more than 7000 working proxies from ~50 sources.
- Support protocols: HTTP(S), SOCKS4/5. Also CONNECT method to ports 80 and 23 (SMTP).
- Proxies may be filtered by type, anonymity level, response time, country and status in DNSBL.
- Work as a proxy server that distributes incoming requests to external proxies. With automatic proxy rotation.
- All proxies are checked to support Cookies and Referer (and POST requests if required).
- Automatically removes duplicate proxies.
- Is asynchronous.
Requirements
Installation
To install last stable release from pypi:
$ pip install proxybroker
The latest development version can be installed directly from GitHub:
$ pip install -U git+https://github.com/constverum/ProxyBroker.git
Usage
CLI Examples
Find
Find and show 10 HTTP(S) proxies from United States with the high level of anonymity:
$ proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10
Grab
Find and save to a file 10 US proxies (without a check):
$ proxybroker grab --countries US --limit 10 --outfile ./proxies.txt
Serve
Run a local proxy server that distributes incoming requests to a pool of found HTTP(S) proxies with the high level of anonymity:
$ proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High
proxybroker --help
for more information on the options available.proxybroker <command> --help
for more information on a command.Basic code example
Find and show 10 working HTTP(S) proxies:
import asyncio from proxybroker import Broker async def show(proxies): while True: proxy = await proxies.get() if proxy is None: break print('Found proxy: %s' % proxy) proxies = asyncio.Queue() broker = Broker(proxies) tasks = asyncio.gather( broker.find(types=['HTTP', 'HTTPS'], limit=10), show(proxies)) loop = asyncio.get_event_loop() loop.run_until_complete(tasks)
Documentation
https://proxybroker.readthedocs.io/
TODO
- Check the ping, response time and speed of data transfer
- Check site access (Google, Twitter, etc) and even your own custom URL's
- Information about uptime
- Checksum of data returned
- Support for proxy authentication
- Finding outgoing IP for cascading proxy
- The ability to specify the address of the proxy without port (try to connect on defaulted ports)
Contributing
- Fork it: https://github.com/constverum/ProxyBroker/fork
- Create your feature branch:
git checkout -b my-new-feature
- Commit your changes:
git commit -am 'Add some feature'
- Push to the branch:
git push origin my-new-feature
- Submit a pull request!
License
Licensed under the Apache License, Version 2.0
This product includes GeoLite2 data created by MaxMind, available from http://www.maxmind.com.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK