11.10 Explore ML data repositories

Source: Ideas from Computational Physics (Landau, Páez, Bordeianu), §11.9. Pointers to public physics / ML datasets — not a scrape of any challenge rules. Continues 11.9b. Next: 11.11.

You now have neurons, shallow nets, $k$-means, Keras-style Dense layers, and RGB histograms. Apply them to real tables and images outside this sandbox.

Places to look

Resource	Why browse
Deep Learn Physics data challenges	HEP-style deep-learning open data
MLPhysics (UCI)	Physics ML portal
Kaggle TrackML	Particle tracking
Papers with Code — physics datasets	Curated list
Carbon nanotubes (Kaggle)	Materials table
CERN IML public datasets	HEP ML sets
CHAMPS scalar coupling	Molecular properties
Severstal steel defects	Industrial vision

Pick one dataset. Ask: supervised or unsupervised? What features? What Loss? Can a Dense line, $k$-means, or RGB histogram baseline beat chance before a huge net?

Picking a physics / ML dataset (concrete checklist)

Before you download a multi-GB challenge dump, answer these on a one-page note:

Check	What to write down
Task type	Supervised (labels) vs unsupervised (clusters / density) vs ranking
Features $\mathbf x$	Columns / pixels / hits — units, missingness, leakage risk
Target $y$	Continuous (regression), class id, or none
Split	How train / val / test are defined (random? by run? by event?)
Baseline metric	One number a dumb model must beat (MSE, accuracy, AUC, inertia)
Size for day-1	Can you prototype on $\lesssim 10^4$ rows or a few hundred images?

A good first pick: tabular physics with clear columns and a published baseline (binding energy, scalar coupling, TrackML toy subset). Avoid starting with unlabeled vision dumps until you have a metric and a split plan.

Mini project checklist

1. Load a small slice (CSV / images) locally with Pandas / OpenCV.
2. Plot raw distributions (masses, pixels, scores).
3. Freeze the split (e.g. 80/20 stratified, or the challenge’s official folds) — never peek at test labels while tuning.
4. Baseline: linear / $k$-means / channel ratio; record the baseline metric.
5. Optional: one Keras Dense or sklearn classifier; compare to the baseline only on the held-out set.
6. Write one sentence: what the metric means in physics units (e.g. MeV RMS, hit purity).

Takeaway

Step	Fail if you skip it
Features + labels	You train on noise or leakage
Explicit split	You overfit and call it progress
Baseline metric	A huge net that barely beats chance looks “good”
Small day-1 slice	You never iterate

Quiz

Explore datasets

1 / 3

Before training a huge net on a new physics table you should…

Pick an answer to see if you are right.

Next: 11.11 NN code sketches.