datasets

module

v5.0.1-rc.6+incompatible Latest Latest Go to latest Published: May 6, 2019 License: Apache-2.0

Details

source{d} datasets for source code analysis and machine learning on source code (ML on Code).

This repository contains all the needed tools and scripts to reproduce the datasets, as well as the academic papers they may relate to.

Public Git Archive
Size: 3TB
Description: 180k+ top-bookmarked repositories from GitHub, consisting of 54M+ files and ~16 billion lines of code over 450+ programming languages.

Programming Language Identifiers
Size: 1GB
Description: ~49M distinct identifiers extracted from 10+ programming languages.

Manually labelled pairs of files and functions
Size: 250MB
Description: 2k Java file and 600 Java function pairs labeled as similar or different by several programmers.

PR review comments
Size: 1.5GB
Description: 25.3 million GitHub PR review comments since January 2015 till December 2018.

Contributions are very welcome, please see CONTRIBUTING.md and code of conduct.

The tools and scripts are licensed under Apache 2.0, see LICENSE.md.

Path	Synopsis
PublicGitArchive
pga command The pga tool provides a simple way to analyze and download the Public Git Archive dataset.	The pga tool provides a simple way to analyze and download the Public Git Archive dataset.
pga-create
pga-create/cmd/pga-create command
pga/cmd Package cmd contains all of the subcommands available in the pga tool.	Package cmd contains all of the subcommands available in the pga tool.
pga/pga Package pga provides a simple API to access the Public Git Archive repository.	Package pga provides a simple API to access the Public Git Archive repository.
pga/pga/filters Package filters provides a set of filters useful to narrow the list of repositories in Public Git Archive.	Package filters provides a set of filters useful to narrow the list of repositories in Public Git Archive.