Skip to content

Data Product Evolution Strategy

Overview

At AIC Holdings, we've developed a progressive maturity model for data products that maximizes team velocity while maintaining production quality. Rather than forcing all data science work through traditional software development cycles, we enable a three-phase evolution path that allows data scientists to build production-ready solutions without extensive engineering overhead.

The Problem with Traditional Approaches

Most organizations face a common challenge: data scientists build powerful prototypes in Jupyter notebooks, but these never reach production due to the complexity of "enterprise" deployment. The result is wasted insights and frustrated teams.

Traditional bottlenecks: - Data scientists wait months for engineering resources - Prototypes die in notebook purgatory
- Simple analytics require complex microservice architecture - Business value is delayed by unnecessary technical complexity

Our Philosophy: Progressive Maturity, Not Platform Migration

Instead of treating prototypes as "throwaway" work that must be rebuilt in "real" technology, we embrace a continuous evolution model where prototypes naturally mature into production systems without rewriting.

Core Principles

🎯 Single Source of Truth: Supabase serves as our unified data plane across all phases, eliminating data silos and integration complexity.

⚡ Velocity Over Perfection: We optimize for speed of insight delivery, not architectural purity.

👥 Inclusive Development: Data scientists can ship production software without deep DevOps knowledge.

📈 Value-Driven Graduation: Products only move to more complex architectures when business value justifies the cost.

The Three-Phase Evolution Model

graph TD
    A[💡 Phase 1: Experiment] --> B{Business Value?}
    B -->|No| C[Archive/Learn]
    B -->|Yes| D[📊 Phase 2: Productionalize]
    D --> E{Platform Integration Needed?}
    E -->|No| F[✅ Stay in Phase 2]
    E -->|Yes| G[🏢 Phase 3: Platform Integration]

    A -.-> H[Streamlit + Railway<br/>Sandbox Schema]
    D -.-> I[Streamlit + Railway<br/>Analytics Schema<br/>Monitoring + SLAs]
    G -.-> J[Next.js + Supabase<br/>Core Schema<br/>Enterprise Features]

    style F fill:#90EE90
    style C fill:#FFB6C1

Phase 1: Experiment

Purpose: Rapid hypothesis testing and concept validation
Technology: Streamlit + Railway + Supabase (sandbox schema)
Timeline: Days to weeks
Team: Individual data scientist or small team

Characteristics: - Quick iteration cycles - Minimal code review requirements
- Direct database access to sandbox data - Focus on proving value, not perfection

Phase 2: Productionalize

Purpose: Stable, reliable data products for regular business use
Technology: Streamlit + Railway + Supabase (analytics schema)
Timeline: Weeks to months of stable operation
Team: Data scientist + light engineering support

Characteristics: - Proper error handling and monitoring - Defined data contracts and SLAs - User authentication and access controls - Regular maintenance and updates - This is production software - many products stay here permanently

Phase 3: Platform Integration

Purpose: Deep integration with core business systems
Technology: Next.js + Supabase (core schema) + Meridian integration
Timeline: Months of development
Team: Full engineering collaboration

Characteristics: - Customer-facing interfaces - Complex workflows and user management - Enterprise-grade security and compliance - Seamless integration with existing business processes

Technical Architecture

graph TB
    subgraph "Supabase Data Plane"
        direction TB
        SB_CORE[(Core Schema<br/>Production Data)]
        SB_ANALYTICS[(Analytics Schema<br/>Business Intelligence)]
        SB_SANDBOX[(Sandbox Schema<br/>Experiments)]
    end

    subgraph "Phase 1: Experiment"
        ST1[Streamlit Apps<br/>Individual Railway Services]
        ST1 --> SB_SANDBOX
    end

    subgraph "Phase 2: Production"
        ST2[Streamlit Apps<br/>Monitored Railway Services]
        ST2 --> SB_ANALYTICS
        MON[Monitoring & Alerting]
        ST2 --> MON
    end

    subgraph "Phase 3: Platform"
        NEXT[Next.js Components<br/>Meridian Integration]
        NEXT --> SB_CORE
        API[API Proxy Layer]
        NEXT --> API
    end

    subgraph "External Data Sources"
        EXT1[Market Data APIs]
        EXT2[PDF Reports]
        EXT3[Internal Systems]
    end

    EXT1 --> SB_CORE
    EXT2 --> SB_SANDBOX
    EXT3 --> SB_CORE

    SB_CORE --> SB_ANALYTICS
    SB_ANALYTICS --> SB_SANDBOX

Graduation Decision Framework

Not all products should graduate to the next phase. Each transition requires clear business justification.

When to Graduate from Experiment to Production

✅ Graduate when: - Daily active usage by target users (>5 regular users) - Stable data requirements identified - Clear business value demonstrated - Basic error handling implemented

❌ Don't graduate if: - Still in active experimentation phase - Data requirements are shifting frequently
- Usage is sporadic or unclear

When to Graduate from Production to Platform

✅ Graduate when: - Customer-facing functionality required - Complex UI beyond Streamlit's capabilities needed - Deep integration with Meridian workflows essential - Security/compliance requirements exceed Streamlit capabilities - Business value justifies 3-6 months of engineering investment

❌ Stay in Production if: - Internal tool serving specific user group well - Streamlit UI meets all user needs - Integration requirements are minimal - Engineering bandwidth is limited

Key insight: Many of our most successful data products remain permanently in Phase 2. This is a feature, not a bug.

Time Savings Analysis

This approach delivers significant time savings compared to traditional enterprise development:

Traditional Enterprise Approach

gantt
    title Traditional Development Timeline
    dateFormat  YYYY-MM-DD
    section Analysis
    Requirements Gathering    :2024-01-01, 30d
    Architecture Planning     :30d
    section Development  
    Backend Development       :60d
    Frontend Development      :45d
    Integration Testing       :30d
    section Deployment
    Production Deployment     :15d
    User Acceptance Testing   :30d

    Total Time: 240 days

Our Progressive Approach

gantt
    title Progressive Evolution Timeline
    dateFormat  YYYY-MM-DD
    section Phase 1
    Streamlit Prototype      :2024-01-01, 7d
    User Feedback           :7d
    section Phase 2
    Production Streamlit     :14d
    Monitoring Setup        :3d
    section Phase 3 (Optional)
    Next.js Integration     :60d

    Time to Value: 14-28 days

Time to Value Comparison: - Traditional: 8 months to first user value
- Our Approach: 2-4 weeks to production value - Improvement: 6-10x faster delivery

Team Collaboration Benefits

Empowered Data Scientists

  • Direct Impact: Data scientists can ship production software independently
  • Reduced Handoffs: No "throw it over the wall" to engineering teams
  • Faster Feedback: Direct user interaction improves product quality
  • Skill Development: Data scientists learn production software skills naturally

Engineering Team Efficiency

  • Focus on Core Platform: Engineering resources concentrate on Meridian and infrastructure
  • Reduced Maintenance: Self-service data products reduce support burden
  • Strategic Projects: More time for high-impact architectural work
  • Quality Gate: Only proven, valuable products require engineering integration

Business Stakeholder Value

  • Rapid Prototyping: See working solutions in days, not months
  • Iterative Improvement: Continuous refinement based on real usage
  • Cost Efficiency: No upfront engineering investment for unproven concepts
  • Reduced Risk: Failed experiments cost days, not months

Real-World Example: MarginIQ Journey

Phase 1 (Week 1): Data scientist built basic PDF processing prototype - Streamlit interface for margin report upload
- Basic OCR extraction and table display - Stored results in sandbox schema - 3 users testing with sample data

Phase 2 (Weeks 2-8): Evolved to production data product
- Enhanced error handling for PDF processing failures - Added portfolio analysis and risk calculations - Moved to analytics schema with proper data contracts - 15+ daily users across risk and trading teams - Railway monitoring and alerting configured

Phase 3 (Future): Platform integration when needed - If MarginIQ needs to integrate with trading systems - If customer-facing margin analysis becomes required - Would become Next.js component in Meridian platform

Result: Production-ready margin analysis tool delivered in 2 weeks instead of 6 months.

Implementation Guidelines

For Data Scientists

  1. Start Simple: Focus on core functionality, not perfect UI
  2. Validate Early: Get user feedback within first week
  3. Document Decisions: Track what works and what doesn't
  4. Think in Phases: Build for current phase, not future complexity

For Engineering Teams

  1. Provide Infrastructure: Maintain Railway templates and Supabase schemas
  2. Create Guardrails: Define security and data access patterns
  3. Review Graduations: Ensure Phase 3 transitions are justified
  4. Support, Don't Control: Enable data scientist independence

For Product Managers

  1. Embrace Experimentation: Encourage rapid prototyping cycles
  2. Measure Value: Define clear success metrics for each phase
  3. Resist Gold-Plating: Not every tool needs enterprise UI
  4. Plan Resources: Phase 3 projects require dedicated engineering time

Success Metrics

Phase 1 Success: - Time from idea to working prototype < 2 weeks - User feedback collected within first month - Clear value proposition identified

Phase 2 Success:
- Daily active users > 5 - Uptime > 99% - User satisfaction > ⅘ - Maintenance overhead < 2 hours/week

Phase 3 Success: - Seamless integration with existing workflows - Enterprise-grade security and performance - Positive ROI within 6 months of integration

Conclusion

Our three-phase evolution strategy transforms data science from a research function into a product delivery engine. By embracing progressive maturity over traditional enterprise architecture, we deliver business value faster while maintaining production quality.

The key insight is simple: not everything needs to be enterprise-grade from day one. By allowing natural evolution from prototype to production to platform, we optimize for the most important metric - time to business value.

This approach has enabled our data science team to deliver production software at startup velocity while maintaining enterprise reliability. Most importantly, it keeps data scientists focused on solving business problems, not wrestling with deployment complexity.