stock-crawler

module
v0.0.0-...-362d1a5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2024 License: Apache-2.0

README

stock-crawler

Coverage Status

Taiwan stock crawling service

Setup Redis

Start Docker container

$ docker-compose -p redis -f build/docker/redis/docker-compose.yml up
Customize Sentinel

Remember to change the image location inside redis/docker-compose.yml

$ cd build/docker/redis
$ docker build -t {Repo}/redis-sentinel -f Dockerfile .

Setup Kafka

Start Kafka container
$ docker-compose -p kafka -f build/docker/kafka/docker-compose.yml up
Create topics
$ docker exec -it kafka-1 bash

./bin/kafka-topics.sh --bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 --create --topic stakeconcentration-v1 --replication-factor 2 --partitions 3
./bin/kafka-topics.sh --bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 --create --topic dailycloses-v1 --replication-factor 2 --partitions 3
./bin/kafka-topics.sh --bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 --create --topic stocks-v1 --replication-factor 2 --partitions 3
./bin/kafka-topics.sh --bootstrap-server kafka-1:9092,kafka-2:9092,kafka-3:9092 --create --topic threeprimary-v1 --replication-factor 2 --partitions 3

Start Application

Build image for Mac M1
$ make docker-build
Start stock-crawler container
$ docker-compose -p stock-crawler -f build/docker/app/docker-compose.yml up
Environment configuration

Please configure .env under project root folder

  1. Concentration data crawling need to use proxy to prevent rate limiting from source website, recommend to use https://app.webscrapingapi.com, can set WEB_SCRAPING={API_KEY}, or https://proxycrawl.com, set PROXY_CRAWL={API_KEY}

Directories

Path Synopsis
internal
cache/mocks
Package cache is a generated GoMock package.
Package cache is a generated GoMock package.
cronjob/mocks
Package cronjob is a generated GoMock package.
Package cronjob is a generated GoMock package.
kafka/mocks
Package kafka is a generated GoMock package.
Package kafka is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL