Skip to main content

Lab 5 — Open datasets and a leakage-safe ML baseline

📍 Where we are: Part VI · Starting Your Own Research — Chapter 17 of 19. Chapter 16 simulated melt pools from physics; this lab works with measured data from a public benchmark, and Chapter 5 told you why such a model is easy to fool.

The simple version

Imagine a student who sits a mock exam where the questions are the same as the practice sheet, only with the numbers shuffled. A top mark proves little. A fair test uses a question type the student has not practised. In machine learning the "practice sheet" is the training data, and a random split of rows is the shuffled mock exam: repeats of the same experiment land on both sides, so the model is graded on what it has effectively already seen. This lab shows that effect with real laser-melting measurements, then does the fair test.

What this chapter covers​

We pick a real open dataset (NIST AM Bench 2022, melt pool cross-sections on nickel alloy 718), check its licence and size before downloading, load and plot it, and train four baselines under two splits. The same model scores 2.2 µm error under a random split and 40.8 µm when a whole process setting is held out. We then add uncertainty estimates, test extrapolation, and write down what the result cannot claim. Everything is in examples/lab-open-data-and-ml/. All outputs below are from our run on 2026-11-15 (macOS, arm64, Python 3.14.6, numpy 2.5.4, pandas 3.1.1, scikit-learn 1.9.1, scipy 1.18.1, matplotlib 3.11.2).

Problem​

Chapter 5 listed leakage and weak validation as the field's most common failure. Leakage means information from the test set reaches the model through the split: when several rows come from one build, one track or one specimen, putting some in training and some in testing lets the model recognise the experiment instead of learning the process [1]. The fix is to split by the unit that you want to generalise over, such as build, part, track or process setting, which is grouped cross-validation [2][3]. The usual excuse for skipping it is that AM data are scarce. This lab uses a deliberately tiny dataset to show that scarcity makes the honest split more important, not less.

State of the art: choosing the data​

Step 1. Look before you download​

Open AM data exist at several scales (NIST AM Bench, the NIST Public Data Repository, and community repositories such as the Materials Data Facility). Checking licence, size and format first saves hours. NIST publishes a metadata API; inspect_record.py uses only the standard library:

cd examples/lab-open-data-and-ml
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt # took 2 min 4 s here, one time
source .venv/bin/activate
make inspect
title : AM Bench 2022 Measurement Results Data: Optical Microscopy of Laser-scanned Single Tracks and Pads (AMB2022-03)
licence: https://www.nist.gov/open/license
version/modified: 1.0.3 2022-09-13
files : 208, total 10.63 GB
by extension: {'sha256': 104, 'tif': 102, 'xlsx': 1, 'txt': 1}
small file: AMB2022-718-SH1-MeltPool_Cross-Section_Measurement_Results.xlsx 25811 bytes
small file: 2718_README.txt 7956 bytes

The record is 10.63 GB, almost all micrographs. The measurements extracted from those images are one 26 kB spreadsheet, which is all this lab needs. The data README grants use, modification and redistribution with acknowledgement of NIST and a notice on modified works [4], under NIST's open-data licence [5]. The record's own README says "Version 1.0.0" while the API reports 1.0.3; we cite the DOI and the file checksum rather than guess which is current [4]. The measurements and challenge are described by Weaver et al. [6] and the benchmark outcome by Levine et al. [7].

Step 2. Download with a checksum​

make data
downloading AMB2022-718-SH1-MeltPool_Cross-Section_Measurement_Results.xlsx
AMB2022-718-SH1-MeltPool_Cross-Section_Measurement_Results.xlsx 25811 bytes sha256=2cfaac96aaca3dab... OK
2718_README.txt 7956 bytes sha256=8fa24c82e087a43e... (no pinned checksum)

NIST does not publish a checksum for this file, so we pinned the hash we received on 2026-11-15 in fetch_data.py. A later mismatch means NIST revised the file: read the version history before trusting old results.

Step 3. Explore​

make explore # 0.8 s
rows x cols: (228, 11)
plates: {'BP2': 94, 'BP3': 92, 'BP1': 42}
== single tracks (BP1) == 42 rows, 7 process cases
P_W v_mm_s D_um n depth_mean depth_sd width_mean
case
0 285 960 67 6 139.7 1.9 136.3
1.1 285 960 49 6 227.2 3.2 106.2
1.2 285 960 82 6 102.4 1.1 141.7
2.1 285 1200 67 6 109.7 1.7 112.9
2.2 285 800 67 6 176.5 2.6 156.1
3.1 325 960 67 6 166.1 2.0 134.3
3.2 245 960 67 6 116.9 1.2 129.4
pooled within-case sd of depth (um): 2.10

Three facts matter. First, plate BP1 holds single tracks: only 7 distinct process settings (laser power P, scan speed v, beam diameter D), each measured six times (three repeat tracks, each in two cross-sections). Second, the design is one factor at a time around a centre case (0): case 1.x changes the beam, 2.x the speed, 3.x the power; nothing varies two factors together. Third, the repeat scatter is only 2.1 µm, against a spread of 41.9 µm across settings, so any model error far above about 2 µm is the model, not the measurement. Plates BP2 and BP3 are pads: adjacent tracks at one fixed setting (186 rows from 6 pads), and they show a strong alternation of depth with track number:

mean depth by track parity (1 = odd tracks):
direction parity
X 0 158.9
1 164.7
Y 0 126.3
1 173.6

The table does not say why; Weaver et al. report that pad depth is reduced when scan and gas-flow directions are parallel [6], which is one possible reading, not one we tested. explore.py also writes figures/single_tracks.png and figures/pads.png. The code that loads the data (common.py) parses Line 1.1_2 into case 1.1, repeat 2. That parsed case is the group key for everything below.

Why it is hard: the split decides the answer​

Step 4. Four baselines, two splits​

Task A predicts depth from log P, log v and log D. Models: the mean (no skill), a power law (linear regression on logs, 4 parameters), a Gaussian process (GP, which also gives an uncertainty [8]) and a random forest. Each is scored twice: random 7-fold over rows, and leave-one-case-out (LOCO: train on six settings, test on the seventh).

make baseline # 4.6 s
== Task A: depth of single tracks (42 rows, 7 cases) ==
model RMSE random-row split RMSE leave-one-case-out MAE leave-one-case-out
mean 42.2 48.3 41.5
power-law 2.4 3.8 2.9
GP 2.2 40.8 25.5
random forest 2.3 43.4 38.0
spread of depth in the data: sd = 41.9 um; within-case repeat sd = 2.1 um (measurement + repeat noise floor)

Two grids of squares, one square per measurement and one column per process case. On the left a random row split puts test squares in every column and the Gaussian process error is 2.2 micrometres; on the right the whole of case 1.1 is held out, the Gaussian process error is 40.8 micrometres and the power law 3.8. The same data and models under two splits: random rows leak repeats of every setting into training (left), leaving a whole setting out tests what you actually want (right). Original diagram by the authors, created with AI assistance.

Read the table in three ways. (1) Under the random split, every model except the mean looks as good as the measurement noise (2.2-2.4 µm). That is the leak: most repeats of every setting are in training. (2) Under LOCO, the GP and random forest fall to barely better than the no-skill mean (40.8 and 43.4 against 48.3 µm); neither extrapolates in a design where each setting is the only one with its value of some input. (3) The power law survives at 3.8 µm, because its structure is a global trend that one setting cannot break. It is the only model whose score under the honest split is within about two noise-floors of the repeat scatter.

Step 5. Uncertainty, and what it does not cover​

The next block of baseline.py output shows what each left-out case looks like and how reliable the stated uncertainty is:

Power law fitted on all 42 rows: depth ~ P^1.24 * v^-1.17 * D^-1.54 (a fit to 7 settings, not a physical law)

case P_W v D_um measured power-law GP GP_sd
0 285 960 67 139.7 141.2 144.7 3.0
1.1 285 960 49 227.2 232.9 135.2 43.0
1.2 285 960 82 102.4 104.1 156.0 63.0
2.1 285 1200 67 109.7 106.1 114.7 31.4
2.2 285 800 67 176.5 171.7 162.3 11.9
3.1 325 960 67 166.1 165.8 158.6 7.9
3.2 245 960 67 116.9 116.7 116.3 15.7

row-level coverage under leave-one-case-out: 33/42 = 79% (nominal 95%)
empirical 95% quantile of |power-law error| over left-out cases: 6.9 um
power-law leave-one-case-out RMSE, bootstrap over 7 cases: median 3.7, 90% interval 2.5-4.5 um
  • The GP's standard deviation does respond to difficulty: 3.0 µm for the centre case, 43.0 and 63.0 µm for the two beam-diameter extremes. But its 95% intervals cover the held-out rows only 79% of the time. A GP's interval is only as honest as its kernel assumptions; with 7 settings it cannot be validated, and we state that rather than recalibrate it.
  • The 6.9 µm band is the 95th percentile of the absolute LOCO errors of the power law, computed over 42 rows that come from only 7 independent cases. The bootstrap that resamples cases gives a 90% interval of 2.5-4.5 µm for the RMSE, itself an estimate from seven numbers. Treat both as indications.
  • The fitted exponents (depth growing with power and falling with speed and beam size, as one expects) are a summary of 7 settings. Weaver et al. report depth correlating with volumetric energy density while width did not within the studied parameters [6]; our fit is an empirical regression, not a test of that statement.

Extrapolation, the case a user cares about, comes from holding out the deepest pool (case 1.1, the smallest beam):

power-law predicted 232.9 um, measured 227.2 um
GP predicted 135.2 um, measured 227.2 um (GP says +/- 84.3 um)
random forest predicted 139.7 um, measured 227.2 um

The power law extrapolates smoothly in log space and lands within 6 µm; the GP and the forest predict values near the centre of the training data, and the GP at least says it is unsure (+/- 84.3 µm).

Step 6. A second task with weaker signal​

Task B predicts pad-track depth from direction, track number and parity, and compares random rows, leave-one-pad-out and leave-one-plate-out:

model RMSE random-row RMSE leave-one-pad-out RMSE leave-one-plate-out
global mean 19.3 19.7 21.4
mean by direction+parity 8.0 8.8 21.4
linear (4 features) 12.7 15.0 23.2
random forest 7.6 7.7 23.3

Here the random split leaks mildly (8.0 against 8.8 µm). The plate-level split is the strict one: the two X pads come from plate BP2 and the Y pads from BP3, so holding out a plate removes a direction altogether, and every model falls to the no-skill 21.4 µm or worse. Whether direction matters is therefore not answerable with this data; that is a design limit, not a modelling failure.

make test # 11.9 s: checks 228/42/186 rows, that no case is in both train and test, and that two runs match expected/

What the model cannot claim​

  1. Nothing about powder-bed fusion. These are bare-plate single laser tracks on IN718 from one testbed, with no powder, no layers, no scan strategy at part scale [6].
  2. Nothing about unseen combinations. Settings vary one factor at a time. The power law has never seen a high power with a low speed; melting mode can switch from conduction to keyhole, where depth is set by evaporation and can be much deeper [9], so a single power law should not be trusted far outside the sampled settings.
  3. No claim that 3.8 µm is "the model error". It is one number from seven folds, and the folds are not independent in any further sense (one plate, one machine, one session).
  4. No transfer to another machine, laser, alloy or lab. There is only one of each. The 2.1 µm repeat scatter is within one session and says nothing about between-machine reproducibility.
  5. Nothing causal from the pads. Direction is confounded with plate, as shown in Step 6.

When it breaks​

Real failures we hit while building this lab:

  • ModuleNotFoundError: No module named 'common'. Running the scripts with python -I isolates them from their own folder, so local imports fail. Use plain python for your own code; use -I only for code that parses untrusted files.
  • A fresh venv install is slow (2 min 4 s here). Do it once; the rest runs in seconds.
  • NIST's metadata API returned zero results for a q= keyword query, while the searchphrase= parameter worked; inspect_record.py searches by the record id string (mds2-2718). Endpoints can change; the DOI link is the stable route.
  • Checksum mismatch means the file changed upstream. Re-read the README's version history.
  • Last-digit differences in make test after a library upgrade: compare to expected/, do not silently overwrite it.

What to try next​

  1. Add case-level uncertainty by fitting the power law with a Bayesian linear model and checking coverage under LOCO.
  2. Add the laser-coupling dataset (NIST record mds2-3842, 93 kB zip, which uses the same case numbers) and ask whether absorbed power explains the cases the power law misses.
  3. Fit width, not only depth, and see whether the same split conclusion holds.
  4. Repeat the pad analysis with a model that uses the previous track's depth, and see how much a random split flatters it.
  5. Compare your numbers with the challenge outcome in [7] and write down what a model must do to count as validated against AM Bench.

Why it matters​

The skill is portable. Almost every AM data table has rows that share a build, part or setting. If you cannot name the group that your split respects, the headline accuracy has no meaning, and reviewers who read the preface's evidence levels will place the claim at "lab scale, contested".

In the real world​

  • NIST publishes AM Bench measurements openly so that modellers can test predictions blind [7]; this lab used one small slice.
  • Reviews of ML-based science document leakage as a recurring cause of overstated results across fields [1].
  • Grouped cross-validation is a standard tool in the scikit-learn documentation [3].

Key terms​

  • Data leakage — information about the test data reaching the model through the training data or the split, so the score is too optimistic.
  • Grouped cross-validation — splitting so that all rows of a group (build, part, track, setting) are entirely in training or entirely in testing.
  • Leave-one-case-out (LOCO) — grouped cross-validation where each process setting is held out in turn.
  • RMSE — root mean squared error, the typical size of the prediction error in the units of the target (here µm).
  • Noise floor — the scatter between repeat measurements of the same setting; model errors near it cannot be improved by modelling.
  • Gaussian process — a regression model that returns both a prediction and a standard deviation for it.
  • Power law — a model in which the output is a product of inputs raised to fixed exponents, linear after taking logarithms.
  • Extrapolation — predicting outside the range of conditions seen in training.
  • One-factor-at-a-time design — an experiment that varies one input per case; it cannot reveal interactions between inputs.
  • Checksum — a short fingerprint of a file (here SHA-256) used to verify that it has not changed.

Where this leads​

Chapter 18, Research starter kit, turns this lab into a method: how to find the literature behind a dataset like this, read it critically, and set up a project that others can reproduce.