dependencies

Dependencies Parser Package

This package provides comprehensive dependency parsing functionality for multiple package management ecosystems. It extracts dependency information from manifest files and converts them into standardized Package URLs (PURLs) following the Package URL specification.

Supported Ecosystems

Installation

go get github.com/scanoss/scanoss.go/pkg/dependencies

Usage

Basic Usage

package main

import (
    "fmt"
    "github.com/scanoss/scanoss.go/pkg/dependencies"
)

func main() {
    // Create a new dependency parser
    parser := dependencies.NewDependencyParser()

    // Parse a single file
    result, err := parser.ParseFile("path/to/go.mod")
    if err != nil {
        panic(err)
    }

    // Print dependencies
    for _, purl := range result.Purls {
        fmt.Printf("PURL: %s\n", purl.Purl)
        if purl.Requirement != "" {
            fmt.Printf("  Requirement: %s\n", purl.Requirement)
        }
        if purl.Scope != "" {
            fmt.Printf("  Scope: %s\n", purl.Scope)
        }
    }
}

Parsing Multiple Files

parser := dependencies.NewDependencyParser()

files := []string{
    "path/to/package.json",
    "path/to/go.mod",
    "path/to/pom.xml",
}

results, err := parser.ParseFiles(files)
if err != nil {
    panic(err)
}

for _, fileDep := range results.Files {
    fmt.Printf("\nFile: %s\n", fileDep.File)
    fmt.Printf("Dependencies: %d\n", len(fileDep.Purls))
}

Filtering Supported Files

parser := dependencies.NewDependencyParser()

allFiles := []string{
    "README.md",
    "package.json",
    "go.mod",
    "main.go",
    "pom.xml",
}

// Filter only files that have a parser
supportedFiles := parser.FilterFiles(allFiles)
// Returns: ["package.json", "go.mod", "pom.xml"]

Checking File Support

parser := dependencies.NewDependencyParser()

if parser.IsSupportedFile("package.json") {
    fmt.Println("This file is supported!")
}

// Get list of all supported file patterns
patterns := parser.SupportedFiles()
for _, pattern := range patterns {
    fmt.Println(pattern)
}

Data Structures

LocalPurl

Represents a single package URL with metadata:

type LocalPurl struct {
    Purl        string  // Package URL (e.g., "pkg:npm/react@18.0.0")
    Requirement string  // Version requirement (e.g., ">=1.0.0")
    Scope       string  // Dependency scope (e.g., "devDependencies")
}

LocalDependency

Represents dependencies from a single file:

type LocalDependency struct {
    File  string      // File path
    Purls []LocalPurl // List of package URLs
}

LocalDependencies

Collection of dependencies from multiple files:

type LocalDependencies struct {
    Files []LocalDependency
}

Package URL (PURL) Format

All parsers generate standardized Package URLs:

pkg:<type>/<namespace>/<name>@<version>?<qualifiers>#<subpath>

Examples by Ecosystem

Node.js:

pkg:npm/react@18.0.0
pkg:npm/@angular/core@15.0.0

Python:

pkg:pypi/django@4.2.0
pkg:pypi/requests

Java (Maven):

pkg:maven/org.springframework/spring-core@5.3.0
pkg:maven/junit/junit@4.13.2?type=jar

Go:

pkg:golang/github.com/spf13/cobra@v1.10.2

Ruby:

pkg:gem/rails@7.0.0

.NET:

pkg:nuget/Newtonsoft.Json@13.0.1

Parser-Specific Features

Node.js Parser

• package.json: Separates dependencies and devDependencies
• package-lock.json: Handles both v1 (nested) and v2+ (flat) formats
• yarn.lock: Parses v1 format (v2 detection included)
• Supports scoped packages (@angular/core)

Python Parser

• requirements.txt:
  • Filters out URLs, paths, and recursive dependencies
  • Handles operators: ==, >=, ~, !=, etc.
  • Exact versions (==) included in PURL, others in Requirement
• pyproject.toml: Parses dependencies from [tool.poetry.dependencies] section

Maven Parser

• pom.xml:
  • Resolves property variables (${project.version})
  • Extracts qualifiers (type, classifier)
  • Captures scopes (compile, test, etc.)
  • Deduplicates dependencies

Gradle Parser

• build.gradle:
  • Supports compact format: implementation 'group:artifact:version'
  • Supports extended format with group/name/version keys
  • Handles multi-line declarations
  • Removes inline comments

Go Parser

• go.mod: Parses require blocks (single-line and multi-line)
• go.sum: Extracts all dependencies with checksums
• Automatically splits namespace/name from import paths

Ruby Parser

• Gemfile: Extracts gem names (no versions)
• Gemfile.lock:
  • State machine parsing (GEM, PATH, GIT sections)
  • Extracts versions and platform specs
  • Skips PATH dependencies (local gems)

NuGet Parser

• *.csproj: Parses modern PackageReference elements
• packages.config: Parses legacy package elements
• Both extract package ID and version

Utility Functions

The parsers package provides several utility functions:

Path & URL Validation

import "github.com/scanoss/scanoss.go/pkg/dependencies/parsers"

// Check if string is a valid URL
if parsers.IsValidURL("https://example.com") {
    // ...
}

// Check if string is a valid path
if parsers.IsValidPath("./requirements.txt") {
    // ...
}

String Processing

// Remove comments and trim whitespace
line := parsers.PreprocessLine("dependency # comment")
// Returns: "dependency"

// Remove inline comments
clean := parsers.TrimComments("code // comment", "//")
// Returns: "code"

Package Name Parsing

// Split namespace and name
namespace, name := parsers.SplitNamespace("github.com/user/repo")
// Returns: ("github.com/user", "repo")

// Parse scoped packages
namespace, name := parsers.GetScopedPackage("@angular/core")
// Returns: ("@angular", "core")

PURL Construction

// Build a Package URL
purl := parsers.BuildPURL(
    "maven",                    // type
    "org.springframework",      // namespace
    "spring-core",              // name
    "5.3.0",                    // version
    map[string]string{          // qualifiers
        "type": "jar",
    },
)
// Returns: "pkg:maven/org.springframework/spring-core@5.3.0?type=jar"

Deduplication

purls := []parsers.LocalPurl{
    {Purl: "pkg:npm/react@18.0.0"},
    {Purl: "pkg:npm/react@18.0.0"}, // duplicate
    {Purl: "pkg:npm/vue@3.0.0"},
}

unique := parsers.RemoveDuplicates(purls)
// Returns: 2 items

Error Handling

The parser handles errors gracefully:

• ParseFile: Returns error if file can't be read or parsed
• ParseFiles: Logs warnings to stderr but continues processing other files
• Invalid files: Returns empty result instead of failing

Example:

result, err := parser.ParseFile("invalid.json")
if err != nil {
    log.Printf("Failed to parse: %v", err)
}

Testing

Run the test suite:

go test -v ./pkg/dependencies/...

The test suite includes:

• Parser orchestration tests
• Individual parser tests for each ecosystem
• PURL generation tests
• Utility function tests

Architecture

pkg/dependencies/
├── parser.go              # Main orchestrator (DependencyParser)
├── parser_test.go         # Comprehensive test suite
└── parsers/
    ├── types.go           # Type definitions (LocalPurl, LocalDependency, etc.)
    ├── utils.go           # Utility functions
    ├── golang.go          # Go parser (go.mod, go.sum)
    ├── python.go          # Python parser (requirements.txt, pyproject.toml)
    ├── npm.go             # Node.js parser (package.json, package-lock.json, yarn.lock)
    ├── maven.go           # Maven parser (pom.xml)
    ├── gradle.go          # Gradle parser (build.gradle)
    ├── ruby.go            # Ruby parser (Gemfile, Gemfile.lock)
    └── nuget.go           # NuGet parser (*.csproj, packages.config)

Contributing

To add a new parser:

1. Create a new file in parsers/ (e.g., rust.go)
2. Implement the parser function with signature:

   func ParseRust(fileContent []byte, filePath string) (*LocalDependency, error)
   

3. Register it in NewDependencyParser() in parser.go
4. Add tests in parser_test.go

License

Part of the SCANOSS project. See the main repository for license information.