gitlab-zoekt-indexer

gitlab-zoekt

A fast, precise code search solution that integrates Zoekt with GitLab. This project provides the infrastructure for indexing and searching code across GitLab repositories with high performance and accuracy.

Overview

gitlab-zoekt is built on top of Zoekt, a fast code search engine maintained by Sourcegraph. It creates specialized search indexes of GitLab repositories by communicating with Gitaly (GitLab's Git repository storage service) and provides a search API for performing fast, exact and regex searches across indexed code.

Key features:

• Fast and accurate code search across GitLab repositories
• Incremental indexing for efficiency (only updating changed files)
• Scalable architecture supporting distributed searching across multiple nodes

This repository includes three binaries:

• gitlab-zoekt-indexer: Original indexer binary
• gitlab-zoekt-webserver: Original webserver binary
• gitlab-zoekt: New unified binary that can run in either indexer or webserver mode

Compiling the binaries

# Build the original indexer
$ make build
$ ./bin/gitlab-zoekt-indexer
2023/08/14 11:07:29 Usage: ./bin/gitlab-zoekt-indexer [ --version | --index_dir=<DIR> | --path_prefix=<PREFIX> | --listen=:<PORT> ]

# Build the original webserver
$ make build_web

# Build the unified binary
$ make build_unified
$ ./bin/gitlab-zoekt
Usage: ./bin/gitlab-zoekt <command> [options]

Commands:
  indexer     Run in indexer mode
  webserver   Run in webserver mode
  version     Print version information

For command specific help:
  ./bin/gitlab-zoekt <command> -help

Using the unified binary

The unified binary supports both indexer and webserver modes:

# Run in indexer mode
$ ./bin/gitlab-zoekt indexer -index_dir=/data/index -listen=:6060

# Run in webserver mode
$ ./bin/gitlab-zoekt webserver -index_dir=/data/index -listen=:6070

# Show help for a specific mode
$ ./bin/gitlab-zoekt indexer -help
$ ./bin/gitlab-zoekt webserver -help

# Show version information
$ ./bin/gitlab-zoekt version

Running indexer in GDK mode

Original binary

1. Set GDK_DIR env variable (for example, export GDK_DIR="$HOME/projects/gdk").
2. Stop GDK indexer if you have it running via gdk stop gitlab-zoekt-indexer-development.
3. Execute make gdk or make watch-gdk. The latter will restart indexer after every change (requires watchexec).

Unified binary

1. Set GDK_DIR env variable (for example, export GDK_DIR="$HOME/projects/gdk").
2. Stop GDK indexer if you have it running via gdk stop gitlab-zoekt-indexer-development.
3. Execute make gdk_unified. This will run the unified binary in indexer mode.

Running indexer with docker-compose

For trying out zoekt. Not an official installation method.

See example

Running tests

1. Install a suitable docker client
2. Compile the unified binary

   make build_unified
   

3. Run the dependencies:

   docker-compose up
   

4. Run the tests:

   # One time
   make test
   
   # On every change (requires https://github.com/watchexec/watchexec installed)
   make watch-test