GitHub - infinitbyte/gopa: GOPA, a spider written in Go.
source link: https://github.com/infinitbyte/gopa
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
infinitbyte/gopa: GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
GOPA, A Spider Written in Go.
- Light weight, low footprint, memory requirement should < 100MB
- Easy to deploy, no runtime or dependency required
- Easy to use, no programming or scripts ability needed, out of box features
Screenshoot
How to use
Requirements
- Elasticsearch v5.3+
Setup
First of all, get it, two opinions: download the pre-built package or compile it yourself.
Download Pre Built Package
Go to Release page, download the right package for your platform.
Note: Darwin is for Mac
Compile The Package Manually
Requirements
- Golang 1.9+
Supported platform
- Mac/Linux: Run
make build
to build the Gopa. - Windows: Checkout this wiki page - How to build GOPA on windows.
For example:
#apt install golang-go
#brew install golang
mkdir ~/go/src/github.com/infinitbyte/ -p
cd ~/go/src/github.com/infinitbyte/
git clone https://github.com/infinitbyte/gopa.git
cd gopa
make
After a few minutes, you should have:
gopa
, the main program, a single binary.gopa.yml
, main configuration for gopa.
Required Config
Note: Elasticsearch version should >= v5.3
- Enable elastic module in
gopa.yml
, update the elasticsearch's setting:
elasticsearch:
- name: default
enabled: true
endpoint: http://localhost:9200
index_prefix: gopa-
basic_auth:
username: elastic
password: changeme
Start
Besides Elasticsearch, Gopa doesn't require any other dependencies, just simply run ./gopa
to start the program.
Gopa can be run as daemon(Note: Only available on Linux and Mac):
Example
➜ gopa git:(master) ✗ ./bin/gopa --daemon ________ ________ __________ _____ / _____/ \_____ \\______ \/ _ \ / \ ___ / | \| ___/ /_\ \ \ \_\ \/ | \ | / | \ \______ /\_______ /____| \____|__ / \/ \/ \/ [gopa] 0.10.0_SNAPSHOT ///last commit: 99616a2, Fri Oct 20 14:04:54 2017 +0200, medcl, update version to 0.10.0 ///
Also run ./gopa -h
to get the full list of command line options.
Example
➜ gopa git:(master) ✗ ./bin/gopa -h ________ ________ __________ _____ / _____/ \_____ \\______ \/ _ \ / \ ___ / | \| ___/ /_\ \ \ \_\ \/ | \ | / | \ \______ /\_______ /____| \____|__ / \/ \/ \/ [gopa] 0.10.0_SNAPSHOT ///last commit: 99616a2, Fri Oct 20 14:04:54 2017 +0200, medcl, update version to 0.10.0 ///
It's safety to press ctrl+c
stop the current running Gopa, Gopa will handle the rest,saving the checkpoint,
you may restore the job later, the world is still in your hand.
If you are running Gopa
as daemon, you may stop it like this:
kill -QUIT `pgrep gopa`
Configuration
- Search Console
http://127.0.0.1:9000/
- Admin Console
http://127.0.0.1:9000/admin/
Architecture
Who uses it?
You use GOPA and you want to be listed there? Contact me.
License
Released under the Apache License, Version 2.0 .
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK