AI Model Lab · evidence-bounded outcome modeling
A small model that knows where its evidence stops
Machine learning on NASA test records, with a gate in front of it. The model answers only inside the conditions NASA actually tested. Everywhere else it abstains and points you to the closest evidence.
- 41 NASA tests used
- 4 without an established flame
- 15 excluded, each with a reason
- 1 experiment family: BASS-II, ISS glovebox
Not a NASA operational fire-safety model. It predicts an experimental outcome, never a mission or crew risk.
Ask the model
Every request is checked first. You get one of four states: in domain, near domain, out of domain or insufficient evidence. Only the first two receive a number.
✓ In domainflame-established v1.0 · 41 NASA tests
Predicted experimental outcome probability: a flame is established after the ignition attempt
80 %90 % bootstrap interval 72 %–88 %
Not a mission fire risk, not a crew safety probability, not a probability of a spacecraft fire. It describes one kind of NASA glovebox test.
Why did the model say this?
Each bar is that condition's push on the log-odds, compared with the average training test. For a logistic model these contributions are exact, not an approximation.
- Oxygen (%) = 17 %−1.38
- Material is PMMA = no−0.48
Average training test: log-odds 3.27 (96 %).
Domain checks
- ✓GravityMicrogravity: every training test burned aboard the ISS
- ✓MaterialSIBAL fabric: 21 training tests, 3 without an established flame
- ✓Pressure101.3 kPa is ISS-cabin pressure (SIBAL tests state 1 atm; PMMA tests state no pressure)
- ✓Oxygen17 % is inside the training range (16.4–21 %)
- ✓Airflow5 cm/s is inside the training range (4–19 cm/s)
- ✓Local support8 SIBAL fabric tests lie within ±1 pp oxygen and ±3 cm/s airflow
Nearest NASA tests in the training data
Distance is a MicroFire heuristic: oxygen and airflow differences in training standard deviations, same material first. It is not a NASA score.
| Test | Material | ΔO₂ | ΔAirflow | NASA recorded |
|---|---|---|---|---|
| GMT178-T12 | ✓ SIBAL fabric | −0.2 pp | +0.0 cm/s | No flame established (Did not ignite) |
| GMT175-T18 | ✓ SIBAL fabric | +0.4 pp | +0.0 cm/s | Flame established (Quenched as flow was reduced) |
| GMT178-T11 | ✓ SIBAL fabric | +0.1 pp | −1.0 cm/s | Flame established (Burned; final outcome not stated in table) |
| GMT178-T14 | ✓ SIBAL fabric | −0.1 pp | −1.0 cm/s | Flame established (Quenched as flow was reduced) |
| GMT178-T17 | ✓ SIBAL fabric | −0.1 pp | +1.0 cm/s | Flame established (Kept burning at high flow (no blowoff)) |
What the data can support
We audited every family before training. Pooling incompatible experiments would raise the record count and lower the science.
| Evidence | Records | Decision | Why |
|---|---|---|---|
| BASS-II SIBAL fabric, concurrent flow | 21 used | Used | Oxygen, starting airflow and 1 atm pressure stated; both outcomes observed; crew sessions recorded, so we can validate across sessions. |
| BASS-II PMMA film, opposed flow | 20 used | Used, with a material term | Same apparatus, gravity and ignition system. Pressure is not stated in the table. Only one PMMA test failed to establish a flame (B14, 14.1 % O₂). |
| BASS-II Nomex | 3 | Excluded | All three failed to sustain a flame. One outcome cannot teach a condition effect. Queries return “insufficient evidence”. |
| Saffire | 20 | Not pooled | Metre-scale samples in a cargo ship at a fixed 20 to 25 cm/s; the outcome is set by material (silicone and Nomex never spread). A different regime from centimetre BASS-II samples. |
| LUCI | 2 | Not pooled | Simulated lunar gravity on a spinning rocket, two burns. Not enough to model, and a different gravity regime. |
What the model predicts
Probability that a flame is established after the ignition attempt in a BASS-II-style ISS glovebox test, as coded from NASA's test-table comments. Coded by MicroFire from NASA's verbatim comments: “no ignition” and “brief flash, no sustained flame” are 0; every test that ignited and spread is 1, however it ended.
Why not “did the flame survive?”
Most BASS-II flames that “went out” were put out on purpose: the crew turned the fan down step by step until the flame quenched, or up until it blew off. That label records the test procedure, so a model trained on it would learn the procedure. We do not train on it.
Excluded records
- GMT45-T1, GMT45-T2, B3, B5NASA marks the oxygen reading as possibly inaccurate
- GMT190-T23, GMT190-T24, GMT216-T25, GMT216-T26, GMT216-T27, GMT216-T28a, GMT216-T28b, GMT216-T29Reused, partly burned sample: NASA notes these give less quantitative data, and reuse changes ignition
- F1, F2, F3Nomex: every test in this family failed to sustain a flame, so a model cannot learn how conditions change the outcome
Benchmark: five models, honest validation
Main scheme: leave one crew session out, so tests from the same day never sit on both sides of the split. Intervals are 95 % bootstrap intervals over the held-out predictions. Gradient boosting was not run: 4 minority-class events cannot support it.
| Model | Brier score ↓ | Log loss ↓ | ROC-AUC | Balanced accuracy at 0.5 | Repeated stratified CV AUC |
|---|---|---|---|---|---|
| Majority-rate baseline (no features) | 0.094 0.03–0.17 | 0.353 | not meaningful | 0.50 0.50–0.50 | n/a |
| Regularised logistic regression: oxygen + material deployed | 0.077 0.03–0.14 | 0.246 | 0.80 0.60–1.00 | 0.50 0.50–0.50 | 0.98 0.97–1.00 |
| Regularised logistic regression: oxygen + airflow + material | 0.080 0.03–0.14 | 0.258 | 0.80 0.60–0.97 | 0.50 0.50–0.50 | 0.93 0.89–0.97 |
| Decision tree, depth 2 | 0.076 0.03–0.14 | 0.257 | 0.71 0.20–1.00 | 0.61 0.46–0.97 | 0.76 0.73–0.78 |
| Random forest, 200 depth-2 trees | 0.074 0.03–0.13 | 0.253 | 0.88 0.70–1.00 | 0.49 0.46–0.50 | 0.91 0.86–0.95 |
How the deployed model was chosen
A model must beat the baseline on grouped Brier score and log loss, and the lower end of its AUC interval must be above 0.5. The simplest model within 0.01 Brier of the best then wins. That is the regularised logistic regression: oxygen + material.
Disclosure: our first draft rule also required balanced accuracy at a 0.5 cut-off. Under that rule the decision tree, depth 2 would have been chosen. We dropped the criterion because the Lab outputs probabilities, never classes, and with 10 % prevalence a 0.5 cut-off is arbitrary. The tree's AUC interval also reaches down to 0.20.
What the numbers really say
The deployed model ranks tests well (AUC 0.80, interval 0.60–1.00) but its Brier interval overlaps the baseline's (0.03–0.17). With 4 failures, it cannot be shown to be better calibrated than always guessing the base rate. It never outputs “no flame” at a 0.5 cut-off, so balanced accuracy at 0.5 stays at chance.
Repeated stratified CV (25 × 4 folds) looks much better (AUC near 0.98) because tests from one session can land on both sides of a split. We report it, but we trust the session-grouped numbers.
Confusion matrix (session-grouped, cut-off 0.5)
| Predicted established | Predicted not | |
|---|---|---|
| Flame established | 37 | 0 |
| No flame | 4 | 0 |
Minority class (no flame): precision n/a, recall 0.00, F1 n/a.
Calibration (session-grouped)
| Predicted range | Tests | Mean prediction | Observed rate |
|---|---|---|---|
| below 0.5 | 0 | – | – |
| 0.5 to 0.8 | 6 | 0.77 | 0.67 |
| 0.8 and above | 35 | 0.94 | 0.94 |
Three bins, because 41 tests cannot support a finer reliability curve. Calibration status: plausible in the 0.8+ bin, not assessable below 0.5.
Stress test: does it transfer to an unseen material?
| Held-out material | Test records | No-flame tests | Brier | AUC | Balanced accuracy |
|---|---|---|---|---|---|
| SIBAL fabric | 21 | 3 | 0.118 | 1.00 | 0.50 |
| PMMA | 20 | 1 | 0.024 | 1.00 | 1.00 |
An oxygen-only model trained on PMMA ranks the fabric tests correctly but calls every one of them “established”. In these tests the fabric failed to ignite at 16.4–16.8 % oxygen, where PMMA films still burned. Ranking transfers; probabilities do not. That is why the gate blocks every material without training data.
Model Card
- Version
- flame-established v1.0
- Purpose
- Show what genuine machine learning on NASA records can support, and where it must stop.
- Predicts
- Probability that a flame is established after the ignition attempt in a BASS-II-style ISS glovebox test, as coded from NASA's test-table comments.
- Does not predict
- Mission fire risk, crew safety, spacecraft fire probability, flame spread rate, extinction, or any outcome in lunar, Martian or Earth gravity, in other atmospheres or for other materials.
- Dataset
- BASS-II test tables (BASS-II Summary Report, Table 7.1, PDF p. 104, BASS-II Summary Report, Table A.1, PDF p. 111): 41 usable tests of 56; 4 without an established flame.
- Features
- Oxygen (volume %) and material (PMMA vs SIBAL), standardised. Starting airflow was benchmarked and did not help.
- Outcome encoding
- 0 = “no ignition” or “brief flash, no sustained flame”; 1 = ignited and spread, however the test ended.
- Missing data
- No imputation. Tests without oxygen or starting airflow would be excluded (none were). Unstated PMMA pressure stays unstated.
- Algorithm
- L2-regularised logistic regression (λ = 1 on standardised features, intercept unpenalised), fitted by Newton's method. Uncertainty: 200 class-stratified bootstrap refits; the interval is their 5th–95th percentile.
- Validation
- Leave-one-crew-session-out (main), 25 × repeated stratified 4-fold, 1,000-sample bootstrap intervals, leave-one-material-out stress test.
- Metrics
- Brier 0.077, log loss 0.246, AUC 0.80 (0.60–1.00). Baseline Brier 0.094.
- Domain rules
- Gravity must be microgravity. Material must be SIBAL fabric or PMMA. Pressure 96.3–106.3 kPa. Oxygen and airflow inside each material's tested range (near domain up to 1 pp and 2 cm/s beyond it). At least 3 same-material tests within ±1 pp and ±3 cm/s.
- Known weaknesses
- Only 4 negative outcomes; one for PMMA. Calibration cannot be shown to beat the base rate. A shared oxygen slope is assumed across two materials and flow configurations. Sessions differ in crew, sample batch and date.
- Gravity, pressure
- Microgravity only. Fabric tests state 1 atm; PMMA tests state no pressure. Nothing here says anything about 56.5 or 70 kPa cabins.
- Potential bias
- Test conditions were chosen by NASA researchers to probe limits, so the data over-represent near-limit oxygen and low flows. That is not a random sample of cabin conditions.
- Reproduce
cd apps/web && npm run model-lab. The pipeline, this page and the tests (lib/model-lab.test.ts) run the same seeded code inlib/model-lab.ts.
Not a NASA operational fire-safety model. Not endorsed by NASA.