SQL Graph Visualizer

Status: Active Development - This project is under active development. APIs may change.

A powerful Go application that transforms SQL database structures (MySQL, PostgreSQL, Oracle, SQL Server) into Neo4j graph databases with interactive visualization and comprehensive performance analysis capabilities. Built with Domain Driven Design architecture and featuring a unified CLI, flexible transformation rules, advanced performance benchmarking, and robust database connection management.

---

Quick Start • Live Demo • Documentation • Discussions • Issues

---

Table of Contents

• Features
• Architecture
• Quick Start
• Installation
• Configuration
• Transformation Rules
• API Documentation
• Visualization
• Testing
• Docker
• Contributing
• Equity Program
• Roadmap
• License

Features

Database Transformation

• Complete SQL to Neo4j conversion with support for MySQL, PostgreSQL, Oracle, and SQL Server
• Flexible rule-based mapping with custom transformation rules
• Custom SQL query support - transform not just tables, but any SQL query result
• Relationship modeling - define directional logical links between nodes
• Property mapping - map SQL columns to Neo4j node properties
• Aggregation support - create analytical nodes from complex queries

Visualization & Analysis

• Interactive graph visualization using Neovis.js and D3.js
• Real-time data exploration with GraphQL queries
• RESTful API for programmatic access
• Customizable node appearance and relationship styling
• Filter and search capabilities within the graph

Performance Analysis & Benchmarking

• Database performance benchmarking with sysbench and custom SQL query sets
• Live MySQL Performance Schema metrics collection (statements, table I/O, indexes, connections)
• Automated bottleneck detection and hotspot analysis
• Query pattern analysis with optimization suggestions
• Performance regression detection across historical benchmark runs
• Benchmark result persistence with JSON/CSV export and summary reporting
• Real-time performance monitoring via WebSocket with visual graph load mapping

Enterprise Architecture

• Domain Driven Design (DDD) - clean, maintainable codebase
• Layered architecture - domain, application, infrastructure, and interface layers
• Dependency injection with ports and adapters pattern
• Comprehensive logging with structured logging support
• Configuration management with YAML-based rules

Developer Experience

• Docker support for easy deployment
• Comprehensive testing suite
• GitHub Actions CI/CD pipeline
• Detailed documentation and examples
• Issue templates for bug reports and feature requests

Architecture

This project follows Domain Driven Design (DDD) principles with a clean layered architecture:

sql-graph-visualizer/
├── cmd/
│   ├── sql-graph-visualizer/   # Unified CLI entry point (cobra)
│   │   ├── main.go
│   │   └── commands/           # CLI subcommands
│   └── main.go                 # Legacy entry point (deprecated)
├── internal/
│   ├── application/            # Application Layer
│   │   ├── bootstrap/          # App initialization & lifecycle
│   │   ├── ports/              # Interface definitions
│   │   └── services/           # Application services
│   ├── domain/                 # Domain Layer
│   │   ├── aggregates/         # Domain aggregates
│   │   ├── entities/           # Domain entities
│   │   ├── events/             # Domain events
│   │   └── models/             # Domain models
│   ├── infrastructure/         # Infrastructure Layer
│   │   ├── middleware/         # HTTP middleware
│   │   └── persistence/        # Database repositories (MySQL, PostgreSQL, Oracle, MSSQL, Neo4j)
│   └── interfaces/             # Interface Layer
│       └── web/                # Web interface files
├── config/                     # Configuration files
├── docs/                       # Documentation
└── scripts/                    # Utility scripts

Tech Stack

• Language: Go 1.24+
• Source Databases: MySQL 8.0+, PostgreSQL 13+, Oracle 19c+, SQL Server 2017+
• Graph Database: Neo4j 4.4+
• CLI Framework: Cobra with shell completion
• API Layer: GraphQL (gqlgen), REST (Gorilla Mux)
• Frontend: HTML5, JavaScript, Neovis.js
• Configuration: Viper + YAML
• Logging: Logrus with structured logging
• Testing: Testify framework
• Containerization: Docker & Docker Compose
• Performance Tools: sysbench, custom SQL benchmark suites
• Connection Management: Database/sql with connection pooling

Quick Start

Prerequisites

• Go 1.24 or higher
• MySQL 8.0+, PostgreSQL 13+, Oracle 19c+, or SQL Server 2017+
• Neo4j 4.4+ (or use Docker)
• Git

1. Clone and Setup

git clone https://github.com/peter7775/sql-graph-visualizer.git
cd sql-graph-visualizer
go mod tidy

2. Start Neo4j (using Docker)

docker-compose up -d neo4j-test

3. Configure Database Connections

cp config/config.yml.example config/config.yml
# Edit config/config.yml with your database credentials

4. Run the Application

# Build the unified CLI
make build

# Run transformation only
./sql-graph-visualizer transform

# Start full server (transform + web UI + API)
./sql-graph-visualizer serve

# Start with specific config and debug logging
./sql-graph-visualizer serve -c config/config.yml -v

5. Access the Application

• Visualization Interface: http://localhost:3000
• GraphQL Playground: http://localhost:8080/graphql
• REST API: http://localhost:8080/api/*
• Neo4j Browser: http://localhost:7474

Installation

From Source

git clone https://github.com/peter7775/sql-graph-visualizer.git
cd sql-graph-visualizer
make build

Using Docker

docker-compose up -d

Using Go Install

go install github.com/yourusername/sql-graph-visualizer@latest

Configuration

The application uses YAML configuration files. The main configuration file is config/config.yml:

# MySQL Configuration
mysql:
  host: localhost
  port: 3306
  user: username
  password: password
  database: dbname
  max_open_conns: 25
  max_idle_conns: 5
  conn_max_lifetime: 5m

# PostgreSQL Configuration (alternative to MySQL)
postgresql:
  host: localhost
  port: 5432
  user: username
  password: password
  database: dbname
  sslmode: disable
  max_open_conns: 25
  max_idle_conns: 5
  conn_max_lifetime: 5m

neo4j:
  uri: bolt://localhost:7687
  user: neo4j
  password: password

transform_rules:
  - name: "users_to_nodes"
    rule_type: "node"
    source:
      type: "query"
      value: "SELECT * FROM users WHERE is_active = 1"
    target_type: "User"
    field_mappings:
      id: "id"
      username: "username"
      email: "email"

Environment Variables

• LOG_LEVEL: Set logging level (debug, info, warn, error)
• CONFIG_PATH: Path to configuration file (default: config/config.yml)
• PORT: HTTP server port (default: 3000)
• API_PORT: API server port (default: 8080)

Transformation Rules

Transformation rules define how MySQL data is converted to Neo4j. There are two main rule types:

Node Rules

Create Neo4j nodes from MySQL data:

- name: "users_to_nodes"
  rule_type: "node"
  source:
    type: "query"  # or "table"
    value: "SELECT u.*, CONCAT(u.first_name, ' ', u.last_name) as full_name FROM users u"
  target_type: "User"
  field_mappings:
    id: "id"
    username: "username"
    full_name: "name"  # Neo4j property name

Relationship Rules

Create Neo4j relationships between nodes:

- name: "user_team_membership"
  rule_type: "relationship"
  relationship_type: "MEMBER_OF"
  direction: "outgoing"  # outgoing, incoming, or both
  source:
    type: "query"
    value: "SELECT user_id, team_id, role, joined_at FROM team_members"
  source_node:
    type: "User"
    key: "user_id"
    target_field: "id"
  target_node:
    type: "Team"
    key: "team_id"
    target_field: "id"
  properties:
    role: "role"
    joined_at: "joined_at"

Advanced Features

• Custom Aggregations: Create analytical nodes from complex SQL queries
• Conditional Logic: Apply rules based on data conditions
• Property Transformation: Transform data types and formats
• Relationship Properties: Add metadata to relationships

Performance Benchmarking

The application includes comprehensive performance benchmarking capabilities to analyze database performance and optimize graph transformations. Benchmarks are configured under performance.benchmarks in config/config.yml and executed via the Performance Benchmarking API.

Supported Benchmark Tools

sysbench (MySQL/PostgreSQL)

performance:
  benchmarks:
    enabled: true
    default_duration: "2m"
    max_duration: "30m"
    results_dir: "data/performance/benchmarks"
    sysbench:
      executable_path: "/usr/bin/sysbench"
      defaults:
        table_size: 100000
        threads: 4
        time: 120

Supported sysbench test types: oltp_read_write, oltp_read_only, oltp_write_only, oltp_point_select, oltp_insert, oltp_update_index, oltp_update_non_index, oltp_delete, select_random_points, select_random_ranges, bulk_insert.

Custom Query Benchmarks

Define named sets of SELECT/INSERT/UPDATE queries to benchmark against the active source database (DDL and DELETE/TRUNCATE statements are rejected as a safety measure):

performance:
  benchmarks:
    custom_queries:
      - name: "user_relationships"
        description: "Test user-to-team relationship queries"
        duration: "2m"
        threads: 4
        queries:
          - query: "SELECT u.*, t.name FROM users u JOIN team_members tm ON u.id = tm.user_id JOIN teams t ON tm.team_id = t.id WHERE u.is_active = 1"
            weight: 70
            description: "Active user team memberships"
          - query: "SELECT COUNT(*) FROM users u JOIN team_members tm ON u.id = tm.user_id GROUP BY tm.team_id"
            weight: 30
            description: "Team member counts"

Run it with POST /api/performance/benchmarks using "tool": "custom" and "query_set": "user_relationships".

Performance Analysis Features

Automated Bottleneck & Hotspot Detection

• Bottleneck identification from benchmark metrics and slow queries
• Hotspot detection across benchmark history, scored by latency/frequency/resource weight
• Query pattern analysis to group similar queries and flag anti-patterns
• Regression detection comparing the latest run against the previous one

Optimization Suggestions & Reporting

• Automatic optimization suggestions (indexing, query rewrites, schema, configuration)
• Overall performance scoring with a rating per benchmark run
• Summary reports via GET /api/performance/reports/summary combining bottlenecks, hotspots, query patterns, and regressions
• Export persisted benchmark history as JSON or CSV via GET /api/performance/export

Database Connection Management

The application provides robust database connection management with automatic failover, connection pooling, and comprehensive error handling.

Connection Features

Automatic Connection Management

• Connection pooling with configurable limits
• Automatic reconnection on connection failures
• Health checks for database availability
• Graceful degradation when databases are unavailable

Multi-Database Support

# Configure multiple databases
databases:
  primary:
    type: "mysql"  # or "postgresql"
    host: "primary-db.example.com"
    port: 3306
    database: "main_db"
    # Connection pool settings
    max_open_conns: 25
    max_idle_conns: 5
    conn_max_lifetime: "5m"
    conn_max_idle_time: "10m"
  
  secondary:
    type: "postgresql"
    host: "secondary-db.example.com"
    port: 5432
    database: "analytics_db"
    sslmode: "require"
    max_open_conns: 15
    max_idle_conns: 3

Connection Error Handling

• Retry mechanisms with exponential backoff
• Circuit breaker pattern for failing connections
• Detailed error logging with connection diagnostics
• Fallback strategies for multi-database setups

Security Features

• SSL/TLS encryption support for all database types
• Connection string validation to prevent injection
• Credential management with environment variable support
• Connection timeout configuration

Performance Optimization

Connection Pooling Best Practices

connection_pools:
  # Production settings
  production:
    max_open_conns: 50
    max_idle_conns: 10
    conn_max_lifetime: "1h"
    conn_max_idle_time: "15m"
  
  # Development settings
  development:
    max_open_conns: 10
    max_idle_conns: 2
    conn_max_lifetime: "30m"
    conn_max_idle_time: "5m"

Monitoring and Diagnostics

• Connection pool metrics (active, idle, waiting connections)
• Query execution timing and slow query detection
• Database health monitoring with periodic checks
• Performance metrics export to monitoring systems

API Documentation

REST API Endpoints

Core Graph API

# Get application configuration
GET /config

# Get graph data (JSON format: nodes + relationships)
GET /api/graph

# Health check
GET /api/health

# Deployment/debug info
GET /api/debug

Performance Benchmarking API

# List benchmark executions
GET /api/performance/benchmarks

# Start a new benchmark
POST /api/performance/benchmarks
{
  "tool": "sysbench",
  "test_type": "oltp_read_write",
  "duration_seconds": 300,
  "threads": 4,
  "database_type": "mysql"
}
# For custom query sets: {"tool": "custom", "query_set": "user_relationships", "duration_seconds": 120}

# Get benchmark status / results
GET /api/performance/benchmarks/{id}
GET /api/performance/benchmarks/{id}/results

# Stop a running benchmark
POST /api/performance/benchmarks/{id}/stop

# Current Performance Schema snapshot (optionally with graph data)
GET /api/performance/data?include_graph=true

# Persisted benchmark history
GET /api/performance/data/history

# Performance data mapped onto the graph
GET /api/performance/data/graph

# Metrics summaries
GET /api/performance/metrics/summary
GET /api/performance/metrics/tables
GET /api/performance/metrics/queries

# Summarized report: bottlenecks, hotspots, query patterns, regressions, optimizations
GET /api/performance/reports/summary

# Export persisted benchmark history
GET /api/performance/export?format=json   # or format=csv

# Real-time monitoring
GET /api/performance/realtime/status
GET /ws/performance   # WebSocket stream of live graph performance data

GraphQL Schema

The GraphQL endpoint provides a flexible query interface for graph data:

query {
  graph {
    nodes { id label properties }
    relationships { from to type properties }
  }
  nodesByType(type: "User") {
    id
    properties
  }
  node(id: "123") {
    id
    label
    properties
  }
  relationshipsByType(type: "MEMBER_OF") {
    from
    to
    properties
  }
  searchNodes(query: "alice") {
    id
    label
  }
  config {
    neo4j { uri username }
  }
}

mutation {
  transformData
}

subscription {
  graphUpdates {
    nodes { id label }
  }
}

GraphQL Playground: http://localhost:8080/graphql

Performance benchmarking and monitoring are exposed via the REST API; the GraphQL schema currently covers graph data only.

Visualization

The web interface provides an interactive graph visualization:

Features

• Interactive Navigation: Pan, zoom, and drag nodes
• Node Filtering: Filter by node types and properties
• Relationship Highlighting: Highlight specific relationship types
• Search Functionality: Find nodes by name or properties
• Layout Options: Different graph layout algorithms
• Export Capabilities: Export graph data or screenshots

Customization

Customize the visualization by modifying the configuration:

visualization:
  node_colors:
    User: "#4CAF50"
    Team: "#2196F3"
    Project: "#FF9800"
  relationship_colors:
    MEMBER_OF: "#757575"
    LEADS: "#F44336"

Testing

Run All Tests

go test ./...

Run Tests with Coverage

go test -cover ./...

Run Specific Package Tests

go test ./internal/domain/...
go test ./internal/application/...

Integration Tests

# Start test databases
docker-compose -f docker-compose.test.yml up -d

# Run integration tests
go test -tags=integration ./...

Load Testing

# Using included load test script
./scripts/load-test.sh

Docker

Development Setup

# Start all services (MySQL, Neo4j, Application)
docker-compose up -d

# View logs
docker-compose logs -f sql-graph-visualizer

# Stop services
docker-compose down

Production Deployment

# Build production image
docker build -t sql-graph-visualizer:latest .

# Run with production configuration
docker run -d \
  --name sql-graph-visualizer \
  -p 3000:3000 \
  -p 8080:8080 \
  -v $(pwd)/config:/app/config \
  sql-graph-visualizer:latest

Health Checks

The Docker container includes health checks:

docker ps  # Check health status
docker inspect sql-graph-visualizer  # Detailed health info

Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run tests and ensure they pass
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to your branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Code Standards

• Follow Go best practices and idioms
• Maintain DDD architecture principles
• Write comprehensive tests
• Update documentation
• Use conventional commit messages

Issue Templates

We provide issue templates for:

• Bug Reports
• Feature Requests
• Database Connection Issues
• Performance Issues

Equity Program for Contributors

Join the commercial success! We offer equity sharing for qualified contributors.

This project has significant commercial potential and we believe in sharing success with those who help build it.

How It Works

Contribute meaningfully → Earn equity stake → Share in commercial licensing revenue

• Equity Tiers: 0.1% - 2.0% based on contribution impact
• Revenue Sharing: From commercial licensing and enterprise deployments
• Vesting: 50% after 6 months of active contribution, 50% after 12 months
• High-Impact Areas: Core algorithms, enterprise features, performance optimization

Qualification Criteria

Automatic Qualification (0.1% - 0.5% equity):

• Merge 3+ significant PRs (marked with equity-eligible label)
• Resolve complex issues (marked with high-impact label)
• Maintain active contribution for 3+ months

High-Impact Qualification (0.5% - 2.0% equity):

• Lead major feature development
• Contribute breakthrough innovations
• Drive adoption and community growth
• Enterprise client development

Commercial Licensing Context

This project operates under a Dual License model:

• Open Source: Free for non-commercial use (AGPL-3.0)
• Commercial: Paid licensing for enterprise use ($2,500+/year)

Revenue Sources:

• Enterprise software licensing
• SaaS platform integrations
• Custom development contracts
• Support and consulting services

Express Your Interest

Ready to contribute and earn equity? Create a Contributor Intent Issue

Or contact directly: petrstepanek99@gmail.com

---

Important: See CONTRIBUTORS.md for complete equity program terms and legal framework.

Roadmap

Completed

• Basic MySQL to Neo4j transformation
• PostgreSQL support with full feature parity
• Rule-based configuration system
• GraphQL API implementation
• Web-based visualization
• Docker containerization
• CI/CD pipeline
• Performance benchmarking integration (sysbench, custom SQL query sets)
• MySQL Performance Schema live monitoring with statement/table/index/connection metrics
• Automated bottleneck & hotspot detection with optimization suggestions
• Real-time performance dashboard with WebSocket updates and graph load overlays
• Benchmark result persistence with historical reporting and JSON/CSV export
• Robust connection management with pooling and failover
• Multi-database connection support
• Oracle Database Support with full schema discovery
• SQL Server (MSSQL) Support with INFORMATION_SCHEMA + sys.* queries
• Unified CLI with cobra subcommands (transform, serve, check, analyze, config, generate, version)

In Progress

• Predictive performance insights exposed via API (trend/anomaly detection engine implemented, REST endpoint pending)
• Enterprise authentication and authorization

Future Plans

• Reverse Transformation: Neo4j to SQL conversion
• Advanced Analytics: Graph algorithms integration (PageRank, Community Detection)
• Cloud Deployment: Kubernetes manifests and Helm charts
• Machine Learning: Automated optimization recommendations
• Monitoring Integration: Prometheus, Grafana, DataDog
• Plugin System: Custom transformation and analysis plugins
• Multi-tenant SaaS: Cloud-hosted solution
• Streaming Data: Real-time database change detection

Performance

Benchmarks

• Small datasets (< 10k nodes): < 5 seconds
• Medium datasets (10k-100k nodes): < 30 seconds
• Large datasets (100k+ nodes): Configurable batch processing

Optimization Tips

• Use indexed columns in transformation queries
• Configure appropriate batch sizes
• Monitor memory usage during large transformations
• Use connection pooling for high-throughput scenarios

Troubleshooting

Common Issues

Connection Errors

# Test MySQL connection
mysql -h localhost -u username -p

# Test Neo4j connection
cypher-shell -a bolt://localhost:7687

Port Conflicts The application automatically handles port conflicts and will find available ports.

Memory Issues For large datasets, increase the batch size in configuration:

processing:
  batch_size: 1000
  max_memory_mb: 2048

Debug Mode

./sql-graph-visualizer serve -v

PostgreSQL Connection Issues

SSL Connection Problems

# Test SSL connection
psql "postgresql://username:password@localhost:5432/dbname?sslmode=require"

# Disable SSL for development
psql "postgresql://username:password@localhost:5432/dbname?sslmode=disable"

Authentication Issues

# Update pg_hba.conf for password authentication
postgresql:
  host: localhost
  port: 5432
  user: username
  password: password
  database: dbname
  sslmode: disable

Performance Benchmarking Issues

sysbench Not Found

# Install sysbench on Ubuntu/Debian
sudo apt-get install sysbench

# Install on macOS
brew install sysbench

# Verify installation
sysbench --version

Custom Query Benchmark Rejected

# Only SELECT/INSERT/UPDATE statements are allowed in custom query sets.
# DDL (CREATE/DROP/ALTER) and DELETE/TRUNCATE statements are rejected.

Benchmark Permission Errors

# Ensure the database user configured for benchmarking has sufficient
# permissions for the statements used:
# - sysbench OLTP tests need SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, DROP TABLE
# - custom query benchmarks need SELECT, INSERT, UPDATE only

Connection Pool Issues

Too Many Connections

# Reduce connection pool size
connection_pools:
  max_open_conns: 10  # Reduce from default 25
  max_idle_conns: 2   # Reduce from default 5

Connection Timeouts

# Increase timeout values
connection_timeout: "30s"
read_timeout: "60s"
write_timeout: "60s"

License

WARNING IMPORTANT: License Change Notice

This project changed from MIT to Dual License on January 6, 2025.

• Prior clones (before Jan 6, 2025): Continue under MIT License ✅
• New features & innovations: Require Dual License compliance 🔒
• See LEGAL_NOTICE.md for complete details

Current License (From January 6, 2025)

This project is available under a Dual License:

Open Source (AGPL-3.0)

• • FREE for open source projects, educational use, and research
• • Source code must remain open source (copyleft)
• • Perfect for learning, contributing, and non-commercial use

Commercial License

• Required for commercial use, SaaS platforms, and enterprise deployments
• Pricing: Starting at $2,500/year for startups
• Includes: Proprietary use rights, enterprise support, custom development

Commercial licensing required for:

• Database management SaaS platforms
• Enterprise monitoring tools integration
• Commercial database consulting services
• White-label or OEM distributions

Contact: petrstepanek99@gmail.com for commercial licensing

Patent-Pending Innovations

This software contains breakthrough innovations in:

• Database consistency validation through graph transformation
• Performance benchmark integration with visual load mapping
• Automated schema discovery and rule generation

See LICENSE for complete terms.

Community & Support

Connect With Us

• Discussions: GitHub Discussions - Ask questions, share ideas
• Email: petrstepanek99@gmail.com - Direct contact & partnerships
• LinkedIn: Connect for professional networking
• Twitter: Follow for updates and announcements

Community Updates

• Newsletter: Monthly development updates and feature releases
• Blog: Technical deep-dives and case studies
• Webinars: Live demos and Q&A sessions

Show Your Support

If this project helps you, consider:

• Star this repository
• Fork and contribute
• Share with your network
• Sponsor development efforts

Acknowledgments

• Neo4j for the excellent graph database
• Neovis.js for graph visualization
• gqlgen for GraphQL implementation
• All contributors who have helped improve this project

---

Made with love by the SQL Graph Visualizer Team