TerediX

Tech Resource Graph (Discovery & Exploration)

🤔 What is TerediX?

TeReDiX (Tech Resource Discover & Exploration) is a tool to discover tech resources for an organization and explore them as a resource graph. Users can use teredix to create resource graphs that depict the relationships and dependencies between different resources. The tool also allows users to search for specific resources and view details about each resource, such as its name, type, description, and associated tags.

teredix can be a useful tool for organizations looking to manage their technology resources more effectively and gain a better understanding of how those resources are interconnected.

This tools can efficiently collect all the resources and their relevent metadata from different sources and can build relationship between resources to enhance visibility of resources.

📄 Table of Contents

• Technical Architecture
• Getting Started
  • Run Discovery
  • Build the Resource Relationship
  • Explore Resource Graph Visualization
• Usage
• Config file
• Supported Source
  • AWS EC2
  • AWS ECR
  • AWS RDS
  • File System
  • GitHub Repository
• Supported Storage
  • PostgreSQL
• Supported Visualization
  • Cytoscape JS
• Contributing
• License

💻Technical Architecture

+-------------------+       +------------------------+        +------------------------+
| Source 1          |       | Scanner 1              |        | Storage Engine         |
| - Scanner         +------>+ - Scan()               |        | - Prepare()            |
|                   |       |                        |        | - Persist()            |
+-------------------+       +------------------------+        | - Find()               |
                                                                   +------------------------+
+-------------------+       +------------------------+        | - GetResources()       |
| Source 2          |       | Scanner 2              |        | - GetRelations()       |
| - Scanner         +------>+ - Scan()               |        | - StoreRelations()     |
|                   |       |                        |        +------------------------+
+-------------------+       +------------------------+

                           +------------------------+
                           | Processor              |
                           | - Process()            |
                           +------------------------+

                           +------------------------+
                           | Visualize              |
                           | - Render()             |
                           +------------------------+

The architecture consists of several components:

Sources: represent the different sources from which resources can be discovered. Each source has its own scanner to discover resources

Scanners: These are responsible for scanning a particular source and returning a list of resources.

Storage Engine: This component stores the discovered resources. It is responsible for preparing the storage schema, persisting resources, and finding resources based on a filter.

Processor: This component orchestrates the discovery process. It starts all the scanners in parallel and processes the resources as they become available.

Visualizer: This component is responsible for displaying the discovered resources and their relationships. It takes in a display type and renders the resources in that format.

🚀 Getting Started

• To get started, you just need to download the latest binary from latest release.
• Prepare your config file. Here is the example config file.

This tool has three major commands available

📥 Run Discovery

To discover the resources, run the following command.

teredix discover --config {your_config.yaml file}

🔗 Build the resource relationship

If you want to build the relationship between resources based on your relation criteria defined in your config file, run -

teredix relation --config {your_config.yaml file}

📈 Explore Resource Graph Visualization

To display the resource graph in an interactive way, please run the following command.

teredix display --config {your_config.yaml file}

It will show Displaying resource graph at http://localhost:8989. Open your browser and visit the URL.

💻 Usage

➜ teredix --help
Discover and Explore your tech resources

Usage:
   [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  discover    Start discovering resources
  display     Display resource graph
  help        Help about any command
  relation    Build resource relationships

Flags:
  -h, --help      help for this command
  -v, --version   version for this command

Config file

Here is an example config.yaml file. You should create your own config file with all the configuration.

---
organization:
  name: "Acme Tech"
  logo: "https://example.com"
discovery:
  name: "Infrastructure Discovery"
  description: "Some description text"
storage:
  batch_size: 2
  engines:
    postgresql:
      host: "localhost"
      port: 5432
      user: "app"
      password: "pass"
      db: "app"
  default_engine: "postgresql"
source:
  fs_one:
    type: "file_system"
    configuration:
      root_directory: "/some/path"
  fs_two:
    type: "file_system"
    configuration:
      root_directory: "/some/other/path"
  github_repo_sh:
    type: "github_repository"
    configuration:
      token: "xxxx"
      user_or_org: "xxxx"
  aws_s3_one:
    type: "aws_s3"
    configuration:
      access_key: "xxxx"
      secret_key: "xxxx"
      session_token: "xxxx"
      region: "x"
      account_id: "xxx"
  aws_rds_one:
    type: "aws_rds"
    config_from: "aws_s3_one"
  aws_ec2_org:
    type: "aws_ec2"
    config_from: "aws_s3_one"
  aws_ecr_org:
    type: "aws_ecr"
    config_from: "aws_s3_one"
relations:
  criteria:
    - name: "file-system-rule1"
      source:
        kind: "FilePath"
        meta_key: "Root-Directory"
        meta_value: "/some/path"
      target:
        kind: "FilePath"
        meta_key: "Root-Directory"
        meta_value: "/some/other/path"

You can get a complete valid config file in valid_config.yaml

The following fields can be specified for each criteria:

🌐 Supported Source

💻 AWS EC2

Discover all EC2 instances and it's tags

source:
  aws_ec2_first_config:
    type: aws_ec2
    configuration:
      access_key: "xxxx"
      secret_key: "xxxx"
      session_token: "xxxx"
      region: "x"
      account_id: "xxx

Available metadata for AWS EC2 Instance

🐳 AWS ECR

Discover all ECR repositories and it's tags

source:
  aws_ecr_work_config:
    type: aws_ecr
    configuration:
      access_key: "xxxx"
      secret_key: "xxxx"
      session_token: "xxxx"
      region: "x"
      account_id: "xxx

Available metadata for AWS ECR repository

☁ AWS RDS

Discover all RDS instances and it's tags

source:
  aws_rds_one:
    type: aws_rds
    configuration:
      access_key: "xxxx"
      secret_key: "xxxx"
      session_token: "xxxx"
      region: "x"
      account_id: "xxx

Available metadata for AWS RDS Instance

📦 AWS S3

Discover all the S3 buckets from AWS

source:
  my_org_s3_bucket_eu:
    type: aws_s3
    configuration:
      access_key: "xxxx"
      secret_key: "xxxx"
      session_token: "xxxx"
      region: "x"
      account_id: "xxx"

Available metadata for AWS S3 Source

:octocat: GitHub Repository

It will fetch all GitHub repositories with metadata

source:
  my_github_repositories:
    type: github_repository
    configuration:
      token: "xxx"
      user_or_org: "xxxx"

Available metadata for github_repository source

📁 File System

It will scan the file system and generate resource for each file. Required configuration:

source:
  csv_data_sets:
    type: file_system
    configuration:
      root_directory: "/path/to/directory"

Available metadata for file_system source

Supported Storage

🐘 PostgreSQL

To configure the PostgreSQL database, you need to update your configuration as following:

storage:
  batch_size: 2
  engines:
    postgresql:
      host: "localhost"
      port: 5432
      user: "app"
      password: "pass"
      db: "app"
  default_engine: postgresql

storage.batch_size control how many resources should be inserted at once. Because all the scanner run as goroutine and provide the resources as a channel for further processing/storing. So it's recommended to use batch_size to avoid consuming heavy memory load if your organization has so many resources for all the source combined

📈 Supported Visualization

To visualize the resource graph, here is the available visualizer.

Cytoscape JS

Example

🤝 Contributing

Contributions are welcome! Please follow the guidelines outlined in the CONTRIBUTING file.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.