README
¶
gh-sql
GitHub SQL is a tool to fetch data from the GitHub API and store it in a SQL database. The ultimate goal is to have the data locally to perform queries on the dataset.
The tool has the following focuses:
- Being targeted towards storing information about specific repositories.
- Discovery of users or repositories is a non-goal.
- Being able to incrementally update the database, fetching what has changed in a repository using the events API (either through polling, even in long intervals (up to 1 month) or through WebSockets).
- Being able to easily "saved" in a single SQL file for quick recovery and replication.
- Replicating the GitHub REST API in read-only operations, without rate limiting (for integration with existing tools that use the GitHub API, by only changing the API endpoint.)
Check out the cookbook for some more examples.
Status
Still pretty heavily work in progress, and I won't make migration strategies, so feel free to build up a database knowing that it'll likely have to be re-created from scratch when this gets updated.
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
schema
Package schema specifies the ent schema types for gh-sql's database.
|
Package schema specifies the ent schema types for gh-sql's database. |
|
pkg
|
|
|
model
Package model defines some types used within the ent schemas.
|
Package model defines some types used within the ent schemas. |
|
model/internal/jsoncatcher
Package jsoncatcher compares re-emitted, parsed JSON data with its raw counterpart to find fields which have not been correctly parsed.
|
Package jsoncatcher compares re-emitted, parsed JSON data with its raw counterpart to find fields which have not been correctly parsed. |
|
rest
Package rest implements a REST API for gh-sql, mimicking the GitHub REST API.
|
Package rest implements a REST API for gh-sql, mimicking the GitHub REST API. |
|
sync
Package sync implements a system to fetch GitHub repositories concurrently.
|
Package sync implements a system to fetch GitHub repositories concurrently. |
|
sync/internal/synchub
Package synchub provides a data structure to manage:
|
Package synchub provides a data structure to manage: |
Click to show internal directories.
Click to hide internal directories.