From raw counts to an explorer-ready JSON

Start from two raw input files

A raw counts matrix and a GEO sample metadata sheet, both exactly as downloaded, with no pre-processing:

The counts file looks like this (first two rows):

GeneID,GSM1616912,GSM1616913,GSM1616914,...
100287102,2,3,3,...
653635,39,72,64,...

Build the metadata columns the package needs

ibdex-projector only needs four columns: a sample ID matching the counts file, a tissue label, and two optional passthrough fields (disease, cohort) that the explorer shows on the patient card. GEO's metadata doesn't come in that shape, so this short script derives it — this is the only hand-written code in the whole walkthrough, everything after this point is the package's own pipeline:

import pandas as pd

meta = pd.read_csv("GSE66207_sample_metadata.csv")

def map_disease(row):
    d = str(row["char_disease"])
    if "Crohn's Disease" in d:
        return "CD"
    if d.startswith("Normal"):
        return "Control"
    return d

out = pd.DataFrame({
    "sample_id": meta["geo_accession"],
    "tissue": "colon",   # source_name_ch1 is just "Colon" for every sample
    "disease": meta.apply(map_disease, axis=1),
    "cohort": "GSE66207",
})
out.to_csv("gse66207_metadata.csv", index=False)

Result: 20 samples mapped to CD, 13 to Control, all tissue colon.

Run the pipeline

From the command line:

ibdex project \
  --counts gse66207_counts.csv \
  --metadata gse66207_metadata.csv \
  --sample-col sample_id \
  --tissue-col tissue \
  --out gse66207_projection.json

or equivalently from Python:

from ibdex_projector.project import project_counts_to_json

project_counts_to_json(
    "gse66207_counts.csv",
    "gse66207_metadata.csv",
    sample_col="sample_id",
    tissue_col="tissue",
    out="gse66207_projection.json",
)

Internally this resolves every Entrez GeneID to an HGNC symbol, converts to log2(CPM + 1), restricts to the frozen 3,000-gene panel, z-scores and clips, encodes with the frozen CVAE, scores pathways, and places each sample onto the reference UMAP layout — the same six steps documented in full in the README's step-by-step walkthrough.

The real QC report from this run

{
  "total_input_genes": 39376,
  "mapped_genes": 39376,
  "unmapped_genes": 0,
  "duplicated_symbols_after_mapping": 2,
  "unique_symbols_after_collapse": 39374,
  "orientation": "genes_x_samples",
  "n_samples": 33,
  "n_panel_genes": 3000,
  "n_panel_genes_present": 3000,
  "n_panel_genes_missing": 0,
  "missing_panel_genes": []
}

Every one of the 39,376 input Entrez IDs resolved to an HGNC symbol, and all 3,000 panel genes were present — nothing was filled in at the training mean for this cohort.

What came out

33 samples placed onto the reference layout, split across four colon endotypes:

UMAP coordinates landed directly inside the reference cohort's own spread (u1: 7.10–14.37, u2: −0.56–7.12), the expected result for a real colon cohort placed into the frozen reference. One sample's full entry in the output JSON:

{
  "id": "GSM1616912",
  "tissue": "colon",
  "tissue_detail": "colon_unspecified",
  "u1": 9.929125785827637,
  "u2": 4.431132793426514,
  "cluster": "C2",
  "disease": "CD",
  "cohort": "GSE66207",
  "sc": [0.6338, -0.4326, 0.9078, 0.5465, 1.4324, 1.0993, -1.4332,
         -1.0996, -0.9069, 0.5074, 0.0361, -0.5726, -1.7206, 0.5355,
         -2.0327, ...]
}

Load it into the explorer

Open the explorer, click Upload external JSON, and pick the output file. The 33 GSE66207 samples render alongside the reference cohort as gold star/diamond markers, selectable and filterable exactly like the built-in cohorts.