go-crawler-example

go-crawler的爬虫示例。

go-crawler-example

go-crawler

english

中文

Usage

baidu-baike

baidu-image

baidu-tieba

baidu-zhidao

bnu

douban

feixiaohao

levi

nike

youtube

zdic

zhihu

clone

git clone [email protected]:lizongying/go-crawler-example.git my-crawler
cd my-crawler

develop

go run cmd/multi_spider/*.go -c example.yml -n test1 -m once

build

make multi_spider
./releases/multi_spider -c prod.yml -n test1 -m once

update go-crawler

go get -u github.com/lizongying/go-crawler@d66d834
go mod tidy
make

Docker image

ghcr.io

hub.docker.com

build

# cross platform
docker buildx create --use
docker buildx inspect --bootstrap

# https://github.com/docker/buildx/issues/290

# for amd64
docker buildx build --platform linux/amd64 -f ./cmd/example_spider/Dockerfile -t lizongying/go-crawler-example:test .

# for arm64(mac m1)
docker buildx build --platform linux/arm64 -f ./cmd/example_spider/Dockerfile -t lizongying/go-crawler-example:test . --load

run

# once
docker run -d lizongying/go-crawler-example:test -c example.yml -n levi -f TestList -m once

# manual
docker run -p 8090-9090:8090 -d lizongying/go-crawler-example:test -c example.yml