COE Overview

What it is

A Python-based ETL pipeline that ingests incident data from multiple security and operations platforms, normalizes it into a PostgreSQL database, and prepares it for weekly reporting. It handles authentication, data fetching, and schema management via Alembic.

Features

• Aggregates events from Jira, Wiz, CrowdStrike, and Vibranium
• Normalizes disparate data sources into a unified PostgreSQL schema
• Manages database schema changes via Alembic migrations
• Supports asynchronous data ingestion for performance
• Integrates with Kubernetes CronJobs via run ID handoff

Quickstart

pip install -e .
export DATABASE_URL=postgresql+asyncpg://user:pass@localhost/db
export JIRA_API_TOKEN=your_token
python -m coe

Architecture

flowchart TD
    User[User] --> Config[Configuration]
    Config --> Pipeline[Ingestion Pipeline]
    Pipeline --> Jira[Jira API]
    Pipeline --> Wiz[Wiz API]
    Pipeline --> CrowdStrike[CrowdStrike API]
    Pipeline --> Vibranium[Vibranium API]
    Jira --> Pipeline
    Wiz --> Pipeline
    CrowdStrike --> Pipeline
    Vibranium --> Pipeline
    Pipeline --> Database[PostgreSQL Database]
    Pipeline --> Output[Run ID File]

How it's built

Built with Python using SQLAlchemy for async database interactions, Pydantic for configuration management, and Alembic for database migrations. It uses structlog for logging and is designed to run as a Kubernetes CronJob.

How it runs

sequenceDiagram
    participant Main as Main Entry
    participant Config as Settings Loader
    participant Pipeline as Ingestion Pipeline
    participant Sources as External APIs
    participant DB as PostgreSQL
    participant Output as Run ID Writer
    Main->>Config: Load Settings
    Config-->>Main: Return Settings
    Main->>Pipeline: Initialize Session
    Pipeline->>Sources: Fetch Events
    Sources-->>Pipeline: Return Raw Data
    Pipeline->>DB: Normalize and Store
    DB-->>Pipeline: Confirm Write
    Pipeline-->>Main: Return Result
    Main->>Output: Write Run ID

How to apply & reuse

Configure environment variables for source APIs (Jira, Wiz, CrowdStrike, Vibranium) and database connection. Run the CLI entrypoint to execute the ingestion pipeline, which writes a run ID for downstream rendering processes.

At a glance