Tutorial¶
A complete run on a simulated cohort, narrated. Nothing is downloaded and the whole thing takes about a minute.
The cohort is simulated along a branching tree, so at the end there is something to check the result against: we know what shape the embedding should recover.
1. A cohort to work with¶
manifold-genetics acquire synthetic writes everything a run needs — genotypes,
labels, a colormap and a config — into one directory.
In [1]:
Copied!
import tempfile
from pathlib import Path
work = Path(tempfile.mkdtemp())
!manifold-genetics acquire synthetic --out {work}
import tempfile
from pathlib import Path
work = Path(tempfile.mkdtemp())
!manifold-genetics acquire synthetic --out {work}
2026-09-16 02:05:58,727 - manifold_genetics.scaffold - INFO - Wrote a simulated cohort and config to /tmp/tmpn4nl3p50 Wrote /tmp/tmpn4nl3p50/config.yaml Drew the tree the cohort lies along in /tmp/tmpn4nl3p50/dla_tree_ground_truth.png Next: manifold-genetics run /tmp/tmpn4nl3p50/config.yaml --dry-run # print the settings manifold-genetics run /tmp/tmpn4nl3p50/config.yaml # do the work
Those are the four kinds of input the pipeline takes:
In [2]:
Copied!
for path in sorted(work.rglob('*')):
if path.is_file():
print(f'{path.relative_to(work)} ({path.stat().st_size:,} bytes)')
for path in sorted(work.rglob('*')):
if path.is_file():
print(f'{path.relative_to(work)} ({path.stat().st_size:,} bytes)')
colormap.json (237 bytes) config.yaml (1,557 bytes) data/fit_subset.bed (250,003 bytes) data/fit_subset.bim (17,783 bytes) data/fit_subset.fam (25,000 bytes) data/labels.csv (34,017 bytes) data/project_subset.bed (500,003 bytes) data/project_subset.bim (17,783 bytes) data/project_subset.fam (50,000 bytes) dla_tree_ground_truth.png (82,738 bytes)
The genotypes are PLINK 1 triples. The labels say which branch of the tree each sample was drawn from:
In [3]:
Copied!
import pandas as pd
labels = pd.read_csv(work / 'data' / 'labels.csv')
print(labels['branch'].value_counts().sort_index().to_string())
labels.head()
import pandas as pd
labels = pd.read_csv(work / 'data' / 'labels.csv')
print(labels['branch'].value_counts().sort_index().to_string())
labels.head()
branch Branch 1 300 Branch 2 300 Branch 3 300 Branch 4 300 Branch 5 100 Branch 6 300 Branch 7 100 Branch 8 300
Out[3]:
| sample_id | branch | |
|---|---|---|
| 0 | SIM0000 | Branch 1 |
| 1 | SIM0001 | Branch 1 |
| 2 | SIM0002 | Branch 1 |
| 3 | SIM0003 | Branch 1 |
| 4 | SIM0004 | Branch 1 |
2. What the config says¶
A run is driven by one YAML file. --dry-run prints every setting it
resolves to, marking (default) on anything the package supplied rather than
the file — the preset contributes most of the embedding parameters.
In [4]:
Copied!
!manifold-genetics run {work}/config.yaml --dry-run
!manifold-genetics run {work}/config.yaml --dry-run
Resolved configuration from /tmp/tmpn4nl3p50/config.yaml:
colormap /tmp/tmpn4nl3p50/colormap.json
embedding phate
embedding_input project
fit_plink /tmp/tmpn4nl3p50/data/fit_subset
labels /tmp/tmpn4nl3p50/data/labels.csv
n_pcs 10
output_dir /tmp/tmpn4nl3p50/outputs
project_plink /tmp/tmpn4nl3p50/data/project_subset
skip_admixture True
skip_admixture_visualization False
skip_embedding False
skip_metrics False
skip_pca False
skip_pca_visualization False
skip_visualization False
admix_batch_size 400 (default)
admix_gpus None (default)
admix_group_column None (default)
admix_threads None (default)
admix_within_group_order chron (default)
fit_colormap None (default)
fit_labels None (default)
geographic_coords None (default)
k_max 10 (default)
k_min 2 (default)
max_fit_memory_gb 8.0 (default)
max_project_memory_gb 8.0 (default)
pca_backend python (default)
project_colormap None (default)
project_labels None (default)
projection_plot_fit_column None (default)
projection_plot_project_column None (default)
embedding_params:
gamma 0
knn 100
n_landmark None
random_landmarking False
t 3
Nothing was run (--dry-run).
3. Run it¶
PCA, then the embedding, then the figures.
In [5]:
Copied!
!manifold-genetics run {work}/config.yaml
!manifold-genetics run {work}/config.yaml
2026-09-16 02:06:18,557 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:18,557 - manifold_genetics.pipeline.orchestrator - INFO - STEP 1: PCA 2026-09-16 02:06:18,557 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:18,557 - manifold_genetics.pipeline.steps.pca - INFO - Running PCA (fit: /tmp/tmpn4nl3p50/data/fit_subset, project: /tmp/tmpn4nl3p50/data/project_subset) 2026-09-16 02:06:18,557 - manifold_genetics.pca.backends.sklearn_backend - INFO - Fitting PCA on 1000 samples x 1000 variants
2026-09-16 02:06:18,757 - manifold_genetics.pca.flashpca - INFO - PCA fitted with 10 components (python backend) 2026-09-16 02:06:18,781 - manifold_genetics.pca.flashpca - INFO - Converted 1000 samples with 10 PCs
2026-09-16 02:06:18,838 - manifold_genetics.pca.flashpca - INFO - Converted 2000 samples with 10 PCs
2026-09-16 02:06:18,865 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:18,865 - manifold_genetics.pipeline.orchestrator - INFO - STEP 1.5: PCA VISUALIZATION 2026-09-16 02:06:18,865 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:18,865 - manifold_genetics.pipeline.steps.viz - INFO - Plotting PCA pairs grid via plot_pca_pairs
2026-09-16 02:06:19,589 - manifold_genetics.visualization.plotting - INFO - Saved PCA pairs plot: /tmp/tmpn4nl3p50/outputs/figures/pca/pca_pairs_by_branch.png 2026-09-16 02:06:19,589 - manifold_genetics.pipeline.steps.viz - INFO - Saved PCA pairs plot: /tmp/tmpn4nl3p50/outputs/figures/pca/pca_pairs_by_branch.png 2026-09-16 02:06:19,589 - manifold_genetics.pipeline.steps.viz - INFO - Created PCA plots: 1 figures 2026-09-16 02:06:19,589 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:19,589 - manifold_genetics.pipeline.orchestrator - INFO - STEP 3: EMBEDDING (PHATE) 2026-09-16 02:06:19,589 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:19,589 - manifold_genetics.pipeline.steps.embedding - INFO - Embedding project PCA coordinates only (mode: project-only) 2026-09-16 02:06:19,599 - manifold_genetics.embeddings.phate - INFO - Fitting PHATE with knn=100, t=3...
2026-09-16 02:06:20,897 - manifold_genetics.embeddings.phate - INFO - ✓ PHATE fitted on 2000 samples 2026-09-16 02:06:20,903 - manifold_genetics.embeddings.phate - INFO - Transforming 2000 samples with PHATE (no batching)...
SGD-MDS may not have converged: stress changed by 2.5% in final iterations. Consider increasing n_iter or adjusting learning_rate.
2026-09-16 02:06:24,568 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:24,568 - manifold_genetics.pipeline.orchestrator - INFO - STEP 4: EMBEDDING VISUALIZATION 2026-09-16 02:06:24,568 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:24,568 - manifold_genetics.pipeline.steps.viz - INFO - Creating project embedding visualizations...
2026-09-16 02:06:24,776 - manifold_genetics.visualization.plotting - INFO - Saved embedding plot: /tmp/tmpn4nl3p50/outputs/figures/embeddings/project_phate_by_branch.png 2026-09-16 02:06:24,776 - manifold_genetics.visualization.plotting - INFO - Generated 1 visualization plots 2026-09-16 02:06:24,776 - manifold_genetics.pipeline.steps.viz - INFO - Created 1 project embedding figures 2026-09-16 02:06:24,776 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.orchestrator - INFO - STEP 5: METRICS 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.orchestrator - INFO - PIPELINE COMPLETE 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.orchestrator - INFO - ====================================================================== 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.orchestrator - INFO - Output directory: /tmp/tmpn4nl3p50/outputs 2026-09-16 02:06:24,777 - manifold_genetics.pipeline.runner - INFO - Pipeline execution complete Pipeline complete. Results in: /tmp/tmpn4nl3p50/outputs
Every stage writes the same shape of file: a sample_id column followed by
dim_1 … dim_n. That is what lets them compose.
In [6]:
Copied!
pcs = pd.read_csv(work / 'outputs' / 'pca' / 'project_pca_10.csv')
emb = pd.read_csv(work / 'outputs' / 'embeddings' / 'phate_2d.csv')
print('PCA ', pcs.shape, list(pcs.columns[:4]))
print('embedding', emb.shape, list(emb.columns))
pcs = pd.read_csv(work / 'outputs' / 'pca' / 'project_pca_10.csv')
emb = pd.read_csv(work / 'outputs' / 'embeddings' / 'phate_2d.csv')
print('PCA ', pcs.shape, list(pcs.columns[:4]))
print('embedding', emb.shape, list(emb.columns))
PCA (2000, 11) ['sample_id', 'dim_1', 'dim_2', 'dim_3'] embedding (2000, 3) ['sample_id', 'dim_1', 'dim_2']
4. Did it work?¶
This is the reason for simulating along a tree. The left panel is the tree the cohort was drawn from — the ground truth. The right is what PHATE recovered from the genotypes alone, having never seen it.
Branches should appear in both, in the same colours. Branches separated by an unsampled gap edge in the tree should come out detached in the embedding.
In [7]:
Copied!
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
truth = mpimg.imread(work / 'dla_tree_ground_truth.png')
found = mpimg.imread(
work / 'outputs' / 'figures' / 'embeddings' / 'project_phate_by_branch.png'
)
fig, axes = plt.subplots(1, 2, figsize=(16, 7))
for ax, image, title in zip(axes, (truth, found), ('Ground truth', 'PHATE embedding')):
ax.imshow(image)
ax.set_title(title, fontsize=15)
ax.axis('off')
plt.tight_layout()
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
truth = mpimg.imread(work / 'dla_tree_ground_truth.png')
found = mpimg.imread(
work / 'outputs' / 'figures' / 'embeddings' / 'project_phate_by_branch.png'
)
fig, axes = plt.subplots(1, 2, figsize=(16, 7))
for ax, image, title in zip(axes, (truth, found), ('Ground truth', 'PHATE embedding')):
ax.imshow(image)
ax.set_title(title, fontsize=15)
ax.axis('off')
plt.tight_layout()
Where to go next¶
- Quickstart — the same commands on a real cohort
- Formats — every file the pipeline reads and writes
- Command line — running the stages individually