Development preview — DSEverse is under active development. Market data and AI-generated responses may be delayed, incomplete, or inaccurate. Do not rely on this website for investment decisions.

DSEverseAnalytics engineering
Checking dbt

Analytics Engineering · Read-only observability

From raw market events to trusted analytics.

A production dbt project turns normalized DSE quotes into session-aware returns, indicators, and reusable marts in ClickHouse—without sitting in the ingestion path.

dbt + ClickHousedbt Core 1.11FastAPIVanilla JavaScript

End-to-end platform

Analytics begins after raw data lands

Kafka keeps transport independent. dbt owns deterministic analytical SQL, tests, lineage, and reusable datasets.

  1. 01DSE ScraperCollect market events
  2. 02KafkaDurable transport
  3. 03Normalizer + SinkValidate and persist
  4. 04ClickHouse RawDurable analytical source
  5. 05dbtTransform and test
  6. 06ProductsAPI, dashboard, ML

Model lineage

One focused transformation graph

Each node has a clear grain and responsibility. Returns and rolling indicators follow observed trading sessions rather than calendar-day joins.

  1. Raw sourcemarket_quotesNormalized quote events
  2. Stagingstg_market_pricesOne session close per symbol and date
  3. Intermediateint_stock_returnsObserved-session returns
  4. Intermediateint_stock_indicatorsRolling price, volatility, and volume signals
  5. Martmart_stock_dailyReusable stock-day dataset
  6. Martmart_market_summaryDaily market breadth and summary

Layered modeling

Built for reuse, not one dashboard

01

Staging

Selects the deterministic market-session close and normalizes the analytical grain to one row per symbol and trade date.

Source truth → analytical truth
02

Intermediate

Calculates returns and rolling indicators with enough prior-session context to keep incremental and full-refresh results equivalent.

Reusable business logic
03

Marts

Publishes stock-day and market-summary datasets for APIs, dashboards, research, and future ML features.

Consumer-ready contracts

Production safeguards

Correct under late data and closed markets

Quality is enforced in the transformation layer while failures remain isolated from ingestion.

Authoritative sessions

Holidays, emergency closures, and special sessions come from the production market calendar.

Partition replacement

Recent month partitions are replaced completely so corrected or removed source keys cannot survive.

Rolling context

Incremental models bring prior observed sessions forward before recalculating returns and indicators.

Data-quality gates

Uniqueness, nullability, price bounds, and current-session freshness are tested on every build.

Single writer

A nonblocking execution lock prevents concurrent jobs from racing analytical tables or artifacts.

Failure isolation

A dbt failure cannot stop scraper, Kafka, normalizer, sink, or raw ClickHouse ingestion.

Execution history

Recent aggregate runs

Only aggregate results are public. Run identifiers, commands, resource names, logs, and failure messages remain private.

CompletedDurationModelsTestsStatus

Loading recent dbt runs…

Public by design

Useful evidence, narrow exposure.

This page reads bounded, sanitized execution summaries from a read-only artifact volume. It cannot invoke dbt, inspect raw logs, query ClickHouse, or mutate the pipeline.