pxier

Pxier -- Simple but powerful free proxy API server

Pxier is an API server based on multiple source of proxy providers, and it's built with Echo, using a MySQL database to store all the data.It also provider a report api for user to report a dead proxy, therefore Pxier could improve the quality in real-time
Pxier's source code is so simply that you can easily fork or pull request. It's designed to keep everything simple.
-- It just a proxy scraper, why bother doing those fascinating abstraction stuff?

Introduction

Pxier is built with four component:

• pxier_fetcher: pxier_fetcher
• pxier_maintainer(optional): pxier_maintainer
• pxier_web: pxier_web
• pxier_db_syncer: pxier_db_syncer

These component's relationship is shown by image below. Here's Pxier's database development:

• At very early version, Pxier use one single pure mysql database, and soon the performance issue came out: When large amount of read&write requests and 10K+ proxies stored, CPU usage will hugely increase.
• After that, Pxier adds cache to the mysql, it's stored as a Map in memory, and sync to database every X seconds.This method hugely lower the CPU usage(100%@10K proxies to 30~60%@30K proxies), but cpu will still spike to the peak when inserting new proxies, delete dead proxy etc.
• Finally, Pxier choose to divide the database to two different part: WriteDB and ReadDB, where WriteDB is the master, and has the most up-to-date data, ReadDB is the slave, it will sync from WriteDB every X seconds. For read Pxier will only query the ReadDB, and for write such as report error proxy, insert new proxy, remove dead proxy, will write the WriteDB. DB syncing is achieved by pxier-db-syncer

The reason not using mysql's replication is to easier deploy in docker. And divide read and write make Pxier much faster and less cpu load

Before start using Pxier

• Pxier only collect proxies, what you plan to do with Pxier is your freedom, not Pxier's business, nor responsible to Pxier
• Pxier is designed to provider an all-in-one solution, for those who being scraped and feel offended, please open an issue, and I'll delete your source right away
• Pixer is an individual project, since it's a scraper, it needs feedback to fix bugs, and more validated source for more "All-In-One"
• Pixer is an individual project, means that I may have no much time for fixing bug, develop more feature, so if you want to get more cool stuff, welcome to contribute or fork then do it yourself
• Pixer is an open source and free project, you're welcome to use, modify, publish your version. But since I don't know that deep about open source license and don't want to spend time for it, I think just put my words here is good enough: If you planned to publish your own fork version, please name the origin author. If you planned to use in a commercial project, you MUST contact me first and MUST NOT use it before I agree.

From Docker-Compose

1. download docker-compose-example.yml and rename it to docker-compose.yml
2. edit your own setting in docker-compose.yml
3. deploy your compose

Build yourself

For every component you will need to do these:

1. git clone the repo
2. cd into the saved directory
3. go mod tidy && go mod vendor && go build -o pixer
4. ./pixer