defrag/ARCHITECTURE.md

# Data Reorganization Architecture: "Project Defrag"

## Executive Summary

This document outlines the architecture for reorganizing 20TB of backup data across multiple NVMe drives and servers. The solution implements intelligent deduplication, systematic categorization, and optimized storage patterns for enhanced performance and maintainability.

## System Architecture Overview

```mermaid
graph TB
    subgraph "Source Environment"
        A["Local Machine<br/>8x NVMe + 1 HDD<br/>~10TB"]
        B["Server Machine<br/>Mixed Storage<br/>~10TB"]
    end
    
    subgraph "Processing Layer"
        C["Discovery Engine"]
        D["Classification Engine"]
        E["Deduplication Engine"]
        F["Migration Engine"]
    end
    
    subgraph "Target Architecture"
        G["App Volumes"]
        H["Gitea Repository"]
        I["Build Cache (.maven, pycache)"]
        J["Artifactories"]
        K["Databases"]
        L["Backups"]
        M["LLM Model Cache"]
        N["Git Infrastructure"]
    end
    
    A --> C
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    F --> H
    F --> I
    F --> J
    F --> K
    F --> L
    F --> M
    F --> N
```

## Data Flow Architecture

### Phase 1: Discovery & Assessment
```mermaid
sequenceDiagram
    participant D as Discovery Engine
    participant FS as File System Scanner
    participant DB as Metadata Database
    participant API as System APIs
    
    D->>FS: Scan directory structures
    FS->>FS: Identify file types, sizes, dates
    FS->>DB: Store file metadata
    D->>API: Query system information
    API->>DB: Store system context
    DB->>D: Return analysis summary
```

### Phase 2: Classification & Deduplication
```mermaid
sequenceDiagram
    participant C as Classifier
    participant DH as Deduplication Hash
    participant CDB as Canonical DB
    participant MAP as Mapping Store
    
    C->>C: Analyze file signatures
    C->>DH: Generate content hashes
    DH->>CDB: Check for duplicates
    CDB->>DH: Return canonical reference
    DH->>MAP: Store deduplication map
    C->>C: Apply categorization rules
```

## Target Directory Structure

```
/mnt/organized/
├── apps/
│   ├── volumes/
│   │   ├── docker-volumes/
│   │   ├── app-data/
│   │   └── user-profiles/
│   └── runtime/
├── development/
│   ├── gitea/
│   │   ├── repositories/
│   │   ├── lfs-objects/
│   │   └── avatars/
│   ├── git-infrastructure/
│   │   ├── hooks/
│   │   ├── templates/
│   │   └── config/
│   └── build-tools/
│       ├── .maven/repository/
│       ├── gradle-cache/
│       └── sbt-cache/
├── artifacts/
│   ├── java/
│   │   ├── maven-central-cache/
│   │   ├── jfrog-artifactory/
│   │   └── gradle-build-cache/
│   ├── python/
│   │   ├── pypi-cache/
│   │   ├── wheelhouse/
│   │   └── pip-cache/
│   ├── node/
│   │   ├── npm-registry/
│   │   ├── yarn-cache/
│   │   └── pnpm-store/
│   └── go/
│       ├── goproxy-cache/
│       ├── module-cache/
│       └── sumdb-cache/
├── cache/
│   ├── llm-models/
│   │   ├── hugging-face/
│   │   ├── openai-cache/
│   │   └── local-llm/
│   ├── pycache/
│   ├── node_modules-archive/
│   └── browser-cache/
├── databases/
│   ├── postgresql/
│   ├── mysql/
│   ├── mongodb/
│   └── redis/
├── backups/
│   ├── system/
│   ├── application/
│   ├── database/
│   └── archive/
└── temp/
    ├── processing/
    ├── staging/
    └── cleanup/
```

## Technology Stack Recommendation

### Primary Language: **Python 3.11+**
**Rationale:**
- Excellent file system handling capabilities
- Rich ecosystem for data processing (pandas, pyarrow)
- Built-in multiprocessing for I/O operations
- Superior hash library support for deduplication
- Cross-platform compatibility

### Key Libraries:
```python
# Core processing
import asyncio
import hashlib
import multiprocessing as mp
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

# Data handling
import pandas as pd
import pyarrow as pa
import sqlite3
import json

# File analysis
import magic  # python-magic
import mimetypes
import filetype

# System integration
import psutil
import shutil
import os
```

## Deduplication Strategy

### Algorithm Selection: **Variable-Size Chunking with Rabin Fingerprinting**

```python
class AdvancedDeduplication:
    def __init__(self, avg_chunk_size=8192):
        self.chunker = RabinChunker(avg_chunk_size)
        self.hash_store = HashStore()
        
    def deduplicate_file(self, file_path):
        chunks = self.chunker.chunk_file(file_path)
        file_hash = self.compute_file_hash(chunks)
        
        if self.hash_store.exists(file_hash):
            return self.create_reference(file_hash)
        else:
            self.store_canonical(file_path, file_hash)
            return file_hash
```

### Performance Optimization:
- **Parallel Processing**: Utilize all CPU cores for hashing
- **Memory Mapping**: For large files (>100MB)
- **Incremental Hashing**: Process files in streams
- **Cache Layer**: Redis for frequently accessed hashes

## Classification Engine

### Rule-Based Classification System:

```yaml
classification_rules:
  build_artifacts:
    patterns:
      - "**/target/**"
      - "**/build/**"
      - "**/dist/**"
      - "**/node_modules/**"
    action: categorize_as_build_cache
  
  development_tools:
    patterns:
      - "**/.maven/**"
      - "**/.gradle/**"
      - "**/.npm/**"
      - "**/.cache/**"
    action: categorize_as_tool_cache
  
  repositories:
    patterns:
      - "**/.git/**"
      - "**/repositories/**"
      - "**/gitea/**"
    action: categorize_as_vcs
  
  database_files:
    patterns:
      - "**/*.db"
      - "**/*.sqlite"
      - "**/postgresql/**"
      - "**/mysql/**"
    action: categorize_as_database
  
  model_files:
    patterns:
      - "**/*.bin"
      - "**/*.onnx"
      - "**/models/**"
      - "**/llm*/**"
    action: categorize_as_ai_model
```

## Performance Considerations

### NVMe Optimization Strategies:

1. **Parallel I/O Operations**
    - Queue depth optimization (32-64 operations)
    - Async I/O with io_uring where available
    - Multi-threaded directory traversal

2. **Memory Management**
    - Streaming processing for large files
    - Memory-mapped file access
    - Buffer pool for frequent operations

3. **CPU Optimization**
    - SIMD instructions for hashing (AVX2/NEON)
    - Process pool for parallel processing
    - NUMA-aware memory allocation

## Migration Strategy

### Three-Phase Approach:

```mermaid
graph LR
    A[Phase 1: Analysis] --> B[Phase 2: Staging]
    B --> C[Phase 3: Migration]
    
    A --> A1[Discovery Scan]
    A --> A2[Deduplication Analysis]
    A --> A3[Space Calculation]
    
    B --> B1[Create Target Structure]
    B --> B2[Hard Link Staging]
    B --> B3[Validation Check]
    
    C --> C1[Atomic Move Operations]
    C --> C2[Symlink Updates]
    C --> C3[Cleanup Verification]
```

## Monitoring & Validation

### Key Metrics:
- **Processing Rate**: Files/second, GB/hour
- **Deduplication Ratio**: Original vs. Final size
- **Error Rate**: Failed operations percentage
- **Resource Usage**: CPU, Memory, I/O utilization

### Validation Checks:
- File integrity verification (hash comparison)
- Directory structure validation
- Symlink resolution testing
- Permission preservation audit

## Risk Mitigation

### Safety Measures:
1. **Read-First Approach**: Never modify source until validation
2. **Incremental Processing**: Process in small batches
3. **Backup Verification**: Ensure backup integrity before operations
4. **Rollback Capability**: Maintain reverse mapping for recovery
5. **Dry-Run Mode**: Preview all operations before execution

## Implementation Timeline

### Phase 1: Tool Development (2-3 weeks)
- Core discovery engine
- Classification system
- Basic deduplication
- Testing framework

### Phase 2: Staging & Validation (1-2 weeks)
- Target structure creation
- Sample data processing
- Performance optimization
- Safety verification

### Phase 3: Production Migration (2-4 weeks)
- Full data processing
- Continuous monitoring
- Issue resolution
- Final validation

This architecture provides a robust, scalable solution for your data reorganization needs while maintaining data integrity and optimizing for your NVMe storage infrastructure.