goebaykleinanzeigen

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 28, 2021 License: MIT Imports: 9 Imported by: 0

README

goebaykleinanzeigen

Disclaimer

This project is in early development and will be buggy.

I am not associated with eBay Kleinanzeigen or eBay in any way.

Use this code at your own risk.

Introduction

goebaykleinanzeigen is an experimental library for scraping eBay Kleinanzeigens data written in go.

eBay Kleinanzeigen should not be confused with the main site eBay.

There is a runnable basic example for scraping car ads in the cmd/simple directory.

This library is written with the intend to scrape car listings but all categories should work and will be added in the future.

Why webscraping?

eBay Kleinanzeigen offers a private API which is only available to their official partners. If you need to use this library you are probably not a official partner.

Sadly this makes accessing eBay Kleinanzeigens data a little lot harder. Especially since we are limited to about 40 req/minute. And of course parsing HTML is pretty error prone.

eBay Kleinanzeigen URLs

In the following we will discuss the relevant URLs for our little scraper

The search URL

First we need to list all the ads.

The basic URL for this may look like this: https://www.ebay-kleinanzeigen.de/l3331r50. Here we can split up the parameter in 2 parts: l3331 and r50

  • Where the 3331 in l3331 is some kind of internal location id for Berlin.
  • Where the 50 in r50 is the distance from the center of the location in kilometers.

This is pretty boring and generic so lets specify a specific category in our URL: https://www.ebay-kleinanzeigen.de/l3331r50c216

  • We added the parameter c216 to our URL! In this case c is probably a prefix for CategoryID and 216 is the ID for cars!

So now we only search for cars but this is still a little bit to generic if you ask me. We don't want to spend too much money on our new car so let's add a specific price range to our query: https://www.ebay-kleinanzeigen.de/preis:1000:4000/l3331r50c216

  • Now we only search for cars in the specific price range from 1000 euro to 4000 euro (preis is the german word for price).

But we don't want a Volkswagen, we only want to search for BMWs. Luckily we can tell eBay to only look for BMWs! Our URL now looks like this: https://www.ebay-kleinanzeigen.de/preis:1000:4000/l3331r50c216+autos.marke_s:bmw.

  • Options specific for a category are appended in the following format: +option_name:option_value

You don't really need to worry about this since the `SearchParam´ struct abstracts the query building away. The SearchParam for this example may look like this:

param := &goebaykleinanzeigen.SearchParam{
		Category:  goebaykleinanzeigen.Cars,
		Location: "3331",
		Radius:    goebaykleinanzeigen.FiftyKM,
		PriceFrom: 1000,
		PriceTo:   4000,
		SpecificParameter: map[goebaykleinanzeigen.ParamName]string{
			goebaykleinanzeigen.CarManufacturer: "bmw",
		},
	}

More specific parameters will be added to specificparams.go in the future.

The ad URL

This is pretty straight forward. The URL for an ad looks like this: https://www.ebay-kleinanzeigen.de/s-anzeige/{ADID}

LocationID for a Location

The internal ID for a location can be fetched by this URL (Berlin in this case): https://www.ebay-kleinanzeigen.de/s-ort-empfehlungen.json?query=Berlin

Ratelimit

Currently the max seems to be about 40 req/minute. This is currently not enforced by the library

Localisation

Keep in mind that eBay Kleinanzeigen is a German Site so everything will be in German.

Prices are integers in euro there are no cents on eBay Kleinanzeigen(? I did not confirm this).

Documentation

Overview

Package goebaykleinanzeigen exposes a webscraping interface for eBay Kleinanzeigen

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AdItem 

type AdItem struct {
	ID              string    `json:"id"`
	ListedSince     time.Time `json:"listed_since"`
	Title           string    `json:"title"`
	Price           int       `json:"price"`
	PriceNegotiable bool      `json:"price_negotiable"`
	Location        string    `json:"location"`
	ZipCode         string    `json:"zip_code"`
	Link            string    `json:"link"`
	Description     string    `json:"description"`
	Details         []*Detail `json:"details"`
	Extras          []string  `json:"extras"`
	Seller          *Seller   `json:"seller"`
}

AdItem represents a single AdItem

type AdListItem 

type AdListItem struct {
	ID              string `json:"id"`
	Title           string `json:"title"`
	Price           int    `json:"price"`
	PriceNegotiable bool   `json:"price_negotiable"`
	Location        string `json:"location"`
	ZipCode         string `json:"zip_code"`
	Link            string `json:"link"`
}

AdListItem a single item in the returned list

type AdListRepo 

type AdListRepo struct {
	// contains filtered or unexported fields
}

AdListRepo represents an AdListRepo The AdListRepo resolves a list of ads

func NewAdListRepo 

func NewAdListRepo(client *http.Client) *AdListRepo

NewAdListRepo creates a new AdRepo, if client is nil, one will be created

func (*AdListRepo) Fetch 

func (al *AdListRepo) Fetch(ctx context.Context, param *SearchParam) (*AdListResponse, error)

Fetch fetches a list of ads based on the provided param

type AdListResponse 

type AdListResponse struct {
	Items      []*AdListItem
	IsLastPage bool
}

AdListResponse represents the response from the Fetch() call

type AdRepo 

type AdRepo struct {
	// contains filtered or unexported fields
}

AdRepo represents an AdRepo The AdRepo resolves a single ad

func NewAdRepo 

func NewAdRepo(client *http.Client) *AdRepo

NewAdRepo creates a new AdRepo, if client is nil, one will be created

func (*AdRepo) Fetch 

func (ar *AdRepo) Fetch(ctx context.Context, url string) (*AdItem, error)

Fetch fetches a single AdItem

type Category 

type Category string

Category represents a specific category id 

const (
	Family         Category = "17"
	Music          Category = "73"
	HouseAndGarden Category = "80"
	Jobs           Category = "102"
	Pets           Category = "130"
	Fashion        Category = "153"
	Electronics    Category = "161"
	Hobby          Category = "185"
	Property       Category = "195"
	Boats          Category = "211"
	Cars           Category = "216"
	CarParts       Category = "223"
	Tickets        Category = "231"
	Education      Category = "235"
	Services       Category = "297"
)

type Detail 

type Detail struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

type LocationID 

type LocationID string

LocationID represents a specifc location retrived from https://www.ebay-kleinanzeigen.de/s-ort-empfehlungen.json?query=Berlin for example

type OfferType 

type OfferType string

OfferType represents the offer type 

const (
	Offer  OfferType = "angebote"
	Wanted OfferType = "gesuche"
)

type ParamName 

type ParamName string

ParamName represents a parameter name for a category specific parameter 

const (
	CarManufacturer       ParamName = "autos.marke_s"
	CarModel              ParamName = "autos.model_s"
	CarKM                 ParamName = "autos.km_i"
	CarYearOfRegistration ParamName = "autos.ez_i"
	CarHP                 ParamName = "autos.power_i"
	CarTUEV               ParamName = "autos.tuevy_i"
)

type Provider 

type Provider string

Provider represents a specific provider (private or commercial seller) 

const (
	Private    Provider = "privat"
	Commercial Provider = "gewerblich"
)

type Radius 

type Radius string

Radius represents the distance from the center of the location 

const (
	WholePlace        Radius = ""
	FiveKM            Radius = "5"
	TenKM             Radius = "10"
	TwentyKM          Radius = "20"
	ThirtyKM          Radius = "30"
	FiftyKM           Radius = "50"
	OneHundredKM      Radius = "100"
	OneHundredFiftyKM Radius = "150"
	TwoHundredKM      Radius = "200"
)

type SearchParam 

type SearchParam struct {
	Category          Category          `json:"category"`
	Provider          Provider          `json:"provider"`
	OfferType         OfferType         `json:"offer_type"`
	Location          LocationID        `json:"location"`
	Radius            Radius            `json:"radius"`
	SpecificParameter SpecificParameter `json:"specific_parameter"`
	// Which page should be scraped
	Page int `json:"page"`
	// Price from in euro
	PriceFrom int `json:"price_from"`
	// Price to in euro
	PriceTo int `json:"price_to"`
}

SearchParam holds all parameters for a search

func ParamsFromJSON 

func ParamsFromJSON(data io.Reader) (*SearchParam, error)

ParamsFromJSON creates a SearchParam struct from JSON

func (*SearchParam) ToJSON 

func (sp *SearchParam) ToJSON(out io.Writer) error

type Seller 

type Seller struct {
	Name         string    `json:"name"`
	ActiveSince  time.Time `json:"active_since"`
	Friendliness string    `json:"friendliness"`
	Rating       string    `json:"rating"`
}

Seller represents the seller of an aditem

type SpecificParameter 

type SpecificParameter map[ParamName]string

SpecificParameter represent specific parameters for each category

Source Files

View all Source files

Directories

Path Synopsis
cmd
multi command
simple command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.