← Back to ProjectsView Architecture

Snyk Vulnerability Dashboard#

Project Overview##

This is a comprehensive vulnerability management system designed to process, analyze, and visualize security vulnerability data from Snyk (a software security platform). The system consists of two main components: a backend data processing script and a web-based dashboard interface. The primary objective is to automate vulnerability reporting, track security trends over time, and provide stakeholders with actionable insights through interactive visualizations and automated email reports.

System Architecture##

Backend Component (Python Script)###

The backend is a Python-based data processing pipeline that:

  • Data Ingestion: Processes Snyk CSV export files containing vulnerability details
  • Data Transformation: Merges vulnerability data with organizational mapping (POD assignments and architect information)
  • Report Generation: Creates daily HTML-formatted reports with trend analysis
  • Communication: Sends automated email reports via Microsoft Outlook integration
  • Change Tracking: Compares current data with previous day's reports to identify new and resolved vulnerabilities
  • JIRA Integration: Capability to create automated tickets for vulnerability tracking (optional)

Frontend Component (Streamlit Dashboard)###

The frontend is a web-based dashboard providing:

  • User Authentication: Cookie-based session management with encrypted storage
  • Multi-page Interface: Five distinct sections for different analytical needs
  • Interactive Visualizations: Charts and graphs for vulnerability trends and distributions
  • Data Export: CSV download functionality for detailed analysis
  • Real-time Analysis: On-demand data fetching and processing

Technical Stack##

Backend Technologies###

  • Python 3.x as the primary programming language
  • pandas for data manipulation and analysis
  • win32com.client for Microsoft Outlook integration
  • JIRA Python library for issue tracking integration
  • sqlite3 for database operations
  • glob and os for file system operations
  • datetime for temporal data handling
  • email.utils for email formatting

Frontend Technologies###

  • Streamlit as the web application framework
  • streamlit_shadcn_ui for enhanced UI components
  • pandas for data processing
  • plotly.express for interactive visualizations
  • pygwalker for advanced data exploration
  • matplotlib and seaborn for statistical plotting
  • sqlite3 for database connectivity
  • streamlit_cookies_manager for session management
  • hashlib for password security

Data Flow and Processing Pipeline##

1. Data Ingestion Process###

  • Reads Snyk vulnerability export files with naming pattern: snyk_issues_detail_{MM_DD_YYYY}*.csv
  • Loads POD mapping file: pod_mapping.csv containing project-to-team assignments
  • Processes data with automatic date-based file selection

2. Data Transformation###

  • Merges vulnerability data with organizational structure (POD and Architect assignments)
  • Handles unmapped projects by creating and maintaining an pod_unmapped.csv file
  • Generates processed output file: snyk_report_{DD-MM-YYYY}.csv
  • Creates unique vulnerability identifiers using package name, problem title, and project target

3. Analysis and Reporting###

  • Compares current day data with previous day for trend analysis
  • Identifies new vulnerabilities and resolved issues
  • Generates comprehensive HTML reports with:
    • Executive summary with total counts
    • Severity breakdown (Critical, High, Medium, Low)
    • POD-wise vulnerability distribution
    • Vulnerability-specific details with auto-fix information
    • Change tracking tables

4. Data Storage###

The system maintains several key data structures in SQLite database:

  • vulnerabilities table: Core vulnerability records with all Snyk data fields
  • daily_summary table: Aggregated daily statistics
  • vulns_update table: Change tracking records
  • users table: Authentication credentials

Database Schema (Inferred Structure)##

Core Tables###

  • vulnerabilities: Contains fields like id, date, issue_severity, problem_title, cve, autofixable, package_name_and_version, project_target, fixed_in_version, pod, architect, problem_id, project_name
  • daily_summary: Stores date, total_issues, unique_vulns, critical_count, high_count, medium_count, low_count
  • vulns_update: Tracks changes with date, increased, resolved flags and vulnerability details
  • users: Authentication table with username and password (hashed)

Feature Set##

Backend Features###

  1. Automated File Processing: Detects and processes latest Snyk export files
  2. Organizational Mapping: Assigns vulnerabilities to specific PODs and architects
  3. Trend Analysis: Compares data across time periods to identify changes
  4. Email Automation: Sends formatted HTML reports to stakeholder groups
  5. Flexible Recipients: Supports both test (self) and production email lists
  6. JIRA Integration: Creates tickets for vulnerability tracking (configurable)
  7. Unmapped Project Tracking: Maintains list of projects without POD assignments

Dashboard Features###

  1. Overview Page:

    • Latest vulnerability summary with day-over-day comparisons
    • Key metrics display using metric cards
    • Severity breakdown visualization

  2. Single-Day Report:

    • Date-specific vulnerability analysis
    • Interactive charts for severity and POD distributions
    • Paginated data tables with column selection
    • CSV export functionality

  3. Period Insights:

    • Date range analysis for trend identification
    • Vulnerability change tracking (increased/resolved)
    • Comparative metrics between start and end dates
    • Time-series visualizations

  4. Visual Analytics:

    • Comprehensive visualization suite
    • Interactive data exploration with PyGWalker
    • Multiple chart types: pie charts, bar charts, line graphs
    • Top vulnerable packages analysis

  5. Authentication System:

    • Secure login with SHA-256 password hashing
    • Session management with encrypted cookies
    • Logout functionality with session cleanup

Configuration and Customization##

Email Configuration###

  • Supports multiple recipient groups (primary and CC lists)
  • HTML email templates with responsive design
  • Attachment support for CSV reports
  • Configurable subject lines with date formatting

JIRA Integration###

  • Configurable server endpoint and authentication
  • Automated ticket creation per POD
  • Customizable issue templates

File Structure###

  • Input Directory: input/ for Snyk CSV files
  • Mapping Directory: mapping/ for organizational data
  • Output Directory: output/ for processed reports
  • Database: snyk_data.db SQLite file

Operational Workflow##

Daily Operations###

  1. Snyk exports vulnerability data to CSV format
  2. Backend script processes new data and generates reports
  3. System compares with previous day's data for change detection
  4. HTML email reports are sent to stakeholders
  5. Dashboard provides real-time access to processed data

User Interactions###

  1. Users authenticate through web interface
  2. Navigate between different analytical views
  3. Select date ranges for specific analysis
  4. Export data for offline analysis
  5. Interact with visualizations for detailed insights

This system provides end-to-end vulnerability management capabilities, from raw data processing to executive-level reporting and detailed technical analysis.

Architecture Diagram

Drag to pan, scroll to zoom