[Free Github Repo] An End-to-End Data Stack on Your Computer

Clone the repo practiceofdata/northstar-data-stack →

Intro

Most tutorials teach one tool. Real pipelines are never one tool.

You learn dbt. You write a model. It works. But you have no idea where the data came from, or what runs it on a schedule, or what happens after.

You learn Airflow. Every DAG prints “hello world.” You learn the syntax. You never move real data.

That’s the gap. Tutorials isolate one tool and treat everything around it as a black box. Real teams work a chain: data lands somewhere, gets transformed, gets tested, gets scheduled, gets reported. Learning one link doesn’t teach you the chain.

So here’s a repo that wires the whole chain together. Free. Open source. Runs on your machine. No cloud account.

It’s called NorthStar Commerce — a fictional e-commerce company’s full analytics stack. Clone it, run one command, and watch raw data turn into a tested, query-ready warehouse.

The stack, four tools, four jobs:

Faker → DuckDB → dbt → Dagster

Faker generates the data. DuckDB stores it. dbt transforms it. Dagster runs it on a schedule. One command — python run_pipeline.py — does all of it in about five minutes.

This isn’t meant to be read once. It’s a sandbox. Run it. Break something on purpose. Modify a model. Re-run it. See what changes.

Business Context

What business is this data even about?

NorthStar Commerce sells consumer electronics online. A mid-sized e-commerce business, asking the questions every e-commerce business asks: What’s our revenue? Which customers are about to churn? What products are driving growth?

Six tables, each standing in for a real source system:

Entity	Table	~Rows	Represents
Customers	customers	5,000	CRM export — tier, location, consent
Products	products	200	Product catalog — price, cost
Orders	orders	~40,000	Order headers
Order line items	order_items	~90,000	Line items per order
Web clickstream	web_events	80,000	Events from an analytics SDK
Marketing campaigns	campaigns	40	Campaign spend and performance

None of it is real. But it isn’t random noise, either. It’s generated with Faker, using a fixed seed. That means two things: it’s reproducible — you and I get the same dataset — and it’s shaped like real behavior: tier-weighted purchases, realistic margins, geographic spread.

Clone the Repo

Here’s how to clone the repo:

git clone https://github.com/practiceofdata/northstar-data-stack

Then make sure you’re inside it:

cd northstar-data-stack

Then go ahead and create a virtual environment, activate it, install the requirements, then run the pipeline script:

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

python run_pipeline.py

That last command runs everything:

• Generates data,
• Loads it into DuckDB,
• Runs dbt models,
• Runs dbt tests,
• Builds the KPI summary

It prints progress as it goes, and when it’s done, you have a live warehouse at warehouse/northstar.duckdb.

After You Run It

You’ll have data populated in your project:

• data/raw/ — six CSVs, ~135,000+ rows total
• warehouse/northstar.duckdb — a live database with three schemas: raw (loaded CSVs), staging (cleaned views), marts_* (finance, marketing, product)
• raw_audit.loads — tracks every load, with row counts and timestamps
• transform/target/ — dbt’s compiled SQL, manifest, test results
• reports/summary.md — an auto-generated KPI summary

You went from nothing to a fact-and-dimension warehouse — revenue rollups, customer lifetime value, product performance — cleaned, joined, and tested. You didn’t write a single line of SQL.

How It Works

The pipeline is a straight line:

Source data → Ingestion → Warehouse → Transform → Test → Orchestrate → Report
  (Faker)      (Python)     (DuckDB)     (dbt)     (dbt)   (Dagster)     (SQL)

Inside the warehouse, dbt organizes tables into three layers:

• Staging (views) — one-to-one with raw sources, renamed, type-cast, lightly filtered
• Intermediate (ephemeral, never materialized) — joins and business logic, like gross profit per order or RFM scores
• Marts (tables) — the final analytics tables, split by domain: finance (fct_orders, mart_daily_revenue), marketing (dim_customers, mart_customer_cohorts), product (dim_products, mart_product_performance)

Dagster wraps this whole flow as a graph of assets — raw data, warehouse load, staging, intermediate, marts, tests, the summary report — each aware of what it depends on, all runnable on a daily schedule.

The Four Tools

Four tools, four jobs. No overlap. That’s why they work together.

Faker generates realistic fake data — names, emails, addresses, prices. It’s seeded for reproducibility and weighted (tiers, purchase frequency, margins) so the output looks like real e-commerce behavior, not noise.

DuckDB is the warehouse. In-process, file-based, columnar. No server, no connection string. Just a file on disk you query directly. Fast for the aggregation-heavy work this stack does. Real teams increasingly use it as a lightweight local warehouse, not a toy.

dbt is the transform layer. You write SQL SELECT statements. dbt turns them into version-controlled, tested, documented tables. It enforces the staging → intermediate → marts pattern, runs schema tests, and builds a lineage graph. The SQL you write here works the same on Snowflake or BigQuery.

Dagster is the orchestrator. It schedules and sequences the pipeline. Airflow thinks in tasks — run this, then that. Dagster thinks in assets — the tables themselves — and models the dependencies explicitly. It knows dim_customers depends on stg_customers, and only re-runs what needs to re-run.

Try These First

Once it’s running, try running these 3 queries.

1. Total revenue

SELECT
    ROUND(SUM(net_revenue), 2)      AS total_revenue,
    COUNT(*)                        AS total_orders,
    ROUND(AVG(net_revenue), 2)      AS avg_order_value
FROM marts_finance.fct_orders
WHERE is_revenue_order;

This hits fct_orders, built by joining orders, line items, and products, with profit computed per order. You didn’t write that join. dbt did.

2. Best customers

SELECT
    full_name, tier, rfm_segment,
    ROUND(lifetime_revenue, 2) AS ltv
FROM marts_marketing.dim_customers
WHERE has_purchased
ORDER BY lifetime_revenue DESC
LIMIT 10;

Every customer is scored on Recency, Frequency, and Monetary value, then labeled — Champion, Loyal, At Risk, Dormant. The logic lives in plain SQL, in transform/models/intermediate/int_customer_metrics.sql. No black box.

3. Product velocity

SELECT
    velocity_tier,
    COUNT(*)                     AS products,
    ROUND(SUM(total_revenue), 2) AS revenue
FROM marts_product.dim_products
GROUP BY 1
ORDER BY revenue DESC;

Products are bucketed by units sold. This shows how concentrated revenue is in a few fast-movers versus a long tail. It usually is.

Go Build Something

You now have a working end-to-end data stack, on your machine. The same pattern — staging to marts, tested transforms, Dagster asset-based orchestration.

Clone it. Run the quick wins. Play with it. Then try these: trace a number back to its raw source, break a test on purpose, watch the Dagster asset graph run, build a new mart.

The repo: github.com/practiceofdata/northstar-data-stack