publiccode-crawler

publiccode.yml crawler for the software catalog of Developers Italia

Description

Developers Italia provides a catalog of Free and Open Source software aimed to Public Administrations.

This crawler retrieves the publiccode.yml files from the repositories of publishers found in the Developers Italia API.

Setup and deployment processes

The crawler can either run manually on the target machine or it can be deployed from a Docker container.

Manually configure and build the crawler

1. Rename config.toml.example to config.toml and set the variables

   NOTE: The application also supports environment variables in substitution to config.toml file. Remember: "environment variables get higher priority than the ones in configuration file"

2. Build the binary with go build

Docker

You can build the Docker image using

docker build .

or use the image published to DockerHub:

docker run -it italia/publiccode-crawler

Commands

crawler crawl

Gets the list of publishers from https://api.developers.italia.it/v1/publishers and starts to crawl their repositories.

crawler crawl publishers*.yml

Gets the list of publishers in publishers*.yml and starts to crawl their repositories.

Other commands

• crawler download-publishers downloads organizations and repositories from the onboarding portal repository and saves them to a publishers YAML file.

See also

• developers-italia-api: the API used to store the results of the crawling
• publiccode-parser-go: the Go package for parsing publiccode.yml files

Authors

Developers Italia is a project by AgID and the Italian Digital Team, which developed the crawler and maintains this repository.