How SabahKu knows what it says

Principles

• Every number on screen has a source and a year. Every modelled number has an uncertainty range and is drawn differently (dashed or hatched).
• Rules before models: the scorecard is a published rule anyone can argue with; models are layered on top, each with a model card.
• Models train on all Malaysian districts so Sabah has enough data and meaningful peers; Sabah results are evaluated leave-one-state-out.
• Associations are never presented as causes. Where evidence is thin, the atlas says so.
• Neutral, non-partisan framing: no statements about individual politicians or parties, and no electoral content.

Sources & vintages

Official statistics are the ground truth. Each dataset is downloaded in bulk (never queried live), stored as an immutable, checksummed snapshot with the publisher's own metadata, and cited with its vintage.

Geography & harmonisation

The atlas maps Sabah's 27 districts on the 2020 boundary layer (geoBoundaries ADM2, CC BY 3.0) and uses all 160 Malaysian districts for context and modelling. Every source name is resolved through a committed alias table; an unknown name stops the pipeline rather than silently failing to join. The alias table covers 193 spellings, e.g. “S.P.Utara”, “Sp Utara” → Seberang Perai Utara; “Nabawan / Persiangan” → Nabawan.

Boundary changes are flagged, not hidden. A value is flagged when the area DOSM reports differs from the polygon it is drawn on:

• Membakut was gazetted out of Beaufort and is reported separately from HIES 2024. Beaufort's 2024 values exclude Membakut, so Beaufort's 2024 trend is not scored. Membakut appears in tables but has no polygon yet.
• Kalabakan is absent from HIES 2019, when it was reported within Tawau; Tawau's 2019 values are flagged.
• Telupid is absent from the 2018–19 Labour Force Survey and 2016 amenities data, when it was within Beluran.
• DOSM reports offshore oil & gas output in a separate “Supra” row not attributable to any district. It is excluded from district rankings and shift-share, and carried separately in forecasts.

Night lights. NASA's Black Marble annual composites (VNP46A4, ≈500 m) are summarised on the same 2020 polygons: water pixels are masked so offshore platforms and fishing fleets don't count, radiance is capped at 500 nW·cm⁻²·sr⁻¹ to limit gas flares, and only good-quality retrievals are used. The atlas reports mean radiance over land pixels with a good-quality retrieval, and flags any district-year where fewer than half the land pixels have one. Lights are a proxy for settlement and electrification, not a measure of output: plantations, mines and offshore fields are dark.

The night map. When the map shows night lights, the picture under the districts is the same masked composite for that year, cropped to Sabah and drawn on one fixed brightness scale (a log scale topping out at 60 nW·cm⁻²·sr⁻¹) so years compare fairly. The district colours and figures come from the table; the picture only shows where inside each district the light is. The pictures are served at /v1/lights.

Indicators

31 indicators. Direction says whether higher is better (↑), worse (↓) or purely descriptive (·). Percentiles are direction-aware so that 100 is always best; descriptive indicators are ranked by value and never called good or bad.

Scorecard rules

Each district is compared with the median of its five structural peers (see typology) in the same year, and with its own trend between the last two observations.

• Strength: above peers by more than the level threshold and not worsening.
• Concern: below peers by more than the level threshold and not improving.
• Mixed: above peers but worsening, or below peers but improving.
• Above/below peers: ±5% for RM values; ±1 percentage point for rates; ±0.01 for Gini.
• Improving/worsening: RM values growing 0.5 pp/yr faster/slower than Sabah as a whole over the same window; rates changing by more than 0.25 pp/yr; Gini by more than 0.002/yr.

Shift-share decomposition

District GDP growth is split into three parts that add up exactly: what the district would have grown at the benchmark's overall rate; an industry-mix effect (was it specialised in fast-growing sectors?); and a competitive effect (did its sectors outperform the same sectors elsewhere?). Benchmarks are Sabah and Malaysia, over 2015–2019 (pre-pandemic, default) and 2015–2020.

NS = E₀·G · IM = E₀·(gᵢ − G) · CE = E₀·(rᵢ − gᵢ) · NS + IM + CE = E₁ − E₀

Typology & structural peers

Ten standardised features for 159 districts (log median income, log GDP per capita, square-root sector shares, log density, share aged 65+, labour participation, piped-water access) are reduced with PCA (6 components, 85% of variance) and clustered with k-means. k = 6 was chosen as the smallest k within 0.01 of the best silhouette (k=6: 0.214, k=7: 0.195, k=8: 0.188, k=9: 0.186). A silhouette near 0.2 means Malaysian districts form a continuum rather than sharp types, so the peers (nearest neighbours) matter more than the cluster labels.

Welfare outcomes (poverty, inequality, unemployment) are deliberately left out of the typology so that peers are structurally similar, and outcomes can then be compared against them.

Naming rules (applied in order to cluster centroids, in z-scores): density > 1 and income > 1 → Metropolitan & industrial core; piped water < −1 → Remote interior, low service access; mining > 1 → Rural economy with mining & quarrying; agriculture > 0.7 and aged 65+ > 0.8 → Ageing agricultural heartland; agriculture > 0.7 → Plantation & agrarian frontier; manufacturing > 1 → Industrial district; services > 0.3 → Services-led towns & suburbs; otherwise Rural mixed economy.

Positive deviance: among a district's ten nearest neighbours, the one whose welfare improved most since 2019 (median income growth %/yr minus change in poverty, pp/yr), with the structural features where it differs most. A lead to investigate, not a prescription.

Driver analysis

How far does a district sit above or below what its structure would typically produce, and which factors are associated with that expectation? Two models are trained on all districts × survey rounds (2019, 2022, 2024) with cross-validation grouped by district. The simpler ridge regression is kept unless LightGBM's error is at least 5% lower.

Nowcasts & projections

GDP. DOSM publishes district GDP to 2020 but Sabah's state GDP by sector to 2025. Each district's 2020 sector mix is grown at Sabah's published sector growth (top-down), adjusted by a shrunk version of its own pre-2020 drift against the state (bottom-up, shrinkage λ = 0.25 chosen on other states' backtests), then reconciled so the districts move exactly with the published state sector totals. Offshore “Supra” output is carried separately. Projections to 2028 extend state sector growth at its non-pandemic median, with transparent scenario shifts.

Night-lights challenger. A second nowcast tilts each district's share of the state total by how fast its night lights grew relative to the state's, with elasticity β chosen on other states' backtests. With β = 0.1, median error at 1/2/3 years is 1.5% / 2.0% / 2.7% against 1.4% / 1.8% / 2.7% for the sector-only model in other states, and 1.5% / 3.9% / 5.2% against 1.5% / 1.9% / 4.8% in Sabah. Every positive β raised the error (year-to-year noise in the lights outweighs their signal at these horizons), so it is reported here but not used; lights remain a descriptive indicator.

Intervals are 80% split-conformal intervals from backtest errors, calibrated on East Malaysian districts (Sabah, Sarawak, W.P. Labuan) because Sabah is more volatile than the peninsula. Honest check: calibrated without Sabah, the intervals covered 74%, 63% and 70% of Sabah outcomes at 1–3 years, below the nominal 80%. That is why the regional calibration is used; with 27 districts these coverage figures are themselves uncertain by about ±8 points. Beyond 3 years, intervals widen with √h and add state-path uncertainty.

Median income. Projection = latest survey value × the state's long-run median-income trend, plus ρ × the district's recent excess growth. The backtest chose ρ = 0: a district's recent excess growth did not help predict its next round (it mean-reverts), so projections follow the state trend. Median error predicting 2024 from 2022: 5.6% nationally, 6.9% in Sabah (out-of-sample coverage 67%). The 2019→2022 backtest spans the pandemic (15.7% error) and is reported, not used.

Scenarios

• Baseline: State sector growth continues at its non-pandemic median.
• Oil & gas downturn: Mining 6 pp/yr slower with a 0.5 pp/yr services spillover. Offshore output ('Supra') bears most of it.
• Tourism & services upswing: Services grow 1.5 pp/yr faster (tourism, logistics).
• Palm-oil downturn: Agriculture grows 4 pp/yr slower (e.g. a sustained CPO price slump or replanting drag).

AI Analyst

The Analyst answers questions and drafts district briefs using read-only tools over this database and a curated document corpus. Every factual sentence must carry a data citation [D#] (indicator, value, source, vintage) or a document citation [R#] (document, page, link). A validator checks that each [D#] exists and that the quoted number matches the database; uncited numbers are removed. Published briefs pass human review; ad-hoc answers are labelled “AI-generated, unreviewed”. See the Analyst page for its evaluation results.

Errata & corrections

• dosm_lfs_district · swk-telang-usan · 2018 · unemployment_rate: u_rate is 293.0 while every other labour-force field for the row is empty; an unemployment rate above 100% is impossible, so the value is treated as a source error. (found 2026-09-26)

Spot a problem? Open an issue on GitHub; corrections are logged here and in the release changelog.

Reproduce everything

The code is open (MIT). One command rebuilds the whole atlas from the public sources:

git clone https://github.com/IlhamKassim/sabah-atlas
cd sabah-atlas && docker compose up -d db && uv sync
uv run atlas build   # ingest → harmonise → gold → models → publish → load

Model versions in this release: drivers-2026.09.26-1642cf15forecast-2026.09.26-74e1d087scorecard-2026.09.26-7683be8fshiftshare-2026.09.26-9f37dafdtypology-2026.09.26-bbb4441b