DATA VISUALIZATION
GITHUB UNIVERSE
An interactive GitHub data visualization universe built with a static-data pipeline. It features a 4-level drill-down (Galaxy -> Country -> Users -> Repositories) driven by deep scraping and query-bucket partitioning.
Overview
Overview
GitHub Universe is an experimental data visualization project that maps public GitHub activity into an explorable cosmos. Using a custom Python scraper with query-bucket partitioning and deep-scrape follower modes, the pipeline statically compiles millions of data points into a high-performance 4-level SVG and WebGL drill-down interface: Countries (Level 0), Cities (Level 1), Users (Level 2), and Repositories (Level 3).
Architecture
Architecture
Data Pipeline
Data Pipeline
Python 3.12
JSON Data Engine
Query-Bucket Partitioning
GitHub Actions
Visualization UI
Visualization UI
SVG Node Rendering
Adaptive FPS HUD
Vercel Deployment
Engineering Decisions
Engineering Decisions
Static-Data Pipeline: Bypasses runtime backend requirements by compiling scraped data into
data/index.jsonand partitioneddata/countries/{CODE}.jsonfiles, achieving zero-latency data serving on Vercel.Query-Bucket Partitioning: The scraper logic segments API requests by 'follower' and 'year' buckets to circumvent top-result API caps, yielding a higher density of data representation per country node.
Adaptive Rendering: Incorporates an automatic performance degradation engine (auto/high/mid/low) with a live FPS HUD, culling off-screen elements via wide-spread layouts and de-crowding algorithms to maintain 60 FPS in dense views.
Mission & Challenges
Mission & Challenges
The Goal: A Cosmic Map
The Goal: A Cosmic Map
The primary objective is to map the vast, invisible landscape of the open-source community into a tangible, explorable cosmos. By treating GitHub not just as a database, but as a digital universe, the visualization allows users to navigate geographically—zooming seamlessly from a macro-galactic view of global countries down into specific cities, and eventually into the individual users (stars) and their repositories (satellites). It transforms raw, tabular API data into an interactive, spatial map of global software engineering density.
The Challenge: Scale & API Limits
The Challenge: Scale & API Limits
The most formidable engineering hurdle was the sheer scale of data retrieval combined with GitHub's strict API rate limits. Scraping a global population sequentially takes an unfeasible amount of time, and the API hard-caps pagination at 1,000 results per query. To bypass this, the scraper was engineered with an advanced "Query-Bucket Partitioning" algorithm. By dynamically slicing search requests into granular sub-buckets (based on exact follower counts and account creation years), the system forces the API to expose deeper layers of hidden users. Additionally, a fault-tolerant state-resume mode was built to handle network drops over continuous multi-day scraping operations.