Doctors read scans, notes, charts and vital signs together. A fusion model does the same: it joins every source into one patient vector, then explains which source drove each call.
Learn how a table, a signal, a note and a scan become numbers, how early and late fusion join them, how a Shapley explanation splits the credit, and how one hospital system put the result in daily use.
Switch the four sources on one at a time. Watch the sick and healthy patients pull apart as each new source joins. Then we build, piece by piece, everything underneath: the numbers for each source, the join, the explanation, and the hospital that used it.
You need a rough idea of what a probability is (a number from 0 to 1 for how likely something is) and a little Python for the code. We build the rest from zero.
Based on Week 3.2 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group. The session’s slides, “HAIM: Holistic AI for Medicine”, are by Dimitris Bertsimas: lecture slides. Every idea is re-taught here in our own words and devices.
Toy cohort, not the lecture’s data: 400 made-up patients (40% sick), seeded so every visit is the same. Each source is informative for a random half of the patients and pure noise for the rest. The fused score adds the switched-on sources; AUC is computed live from all 400.
Chapter 0
See how a doctor combines scans, notes, tables and signals, and why a machine that sees only one of them misses part of the patient
A man in his sixties walks into an emergency room. He is short of breath. A doctor has about ten minutes to decide how worried to be.
She does not look at one thing. She looks at the chest X-ray. She reads the nurse’s note from last night. She scans the table of lab results. She watches the heart rate climb on the monitor. Each source tells her something that the others do not.
Each kind of data is a modality: a form in which information arrives, such as a picture, a sheet of numbers, a line of text or a signal that changes over time. A system that uses more than one modality is multimodal. The act of joining them to make one decision is fusion. Think of a kitchen: each ingredient tastes different alone, and the dish is what you get after you combine them.
The session is a guest lecture on one application, medicine. Its third slide lists what doctors use to decide: scans (MRI, CT, X-ray), language (radiology reports, doctors’ notes, nurses’ notes), tabular data (electronic medical records), and time series and genomic information. Then it asks the question that drives everything after it: can machines use multimodal data to make medical diagnoses and decisions?
Two terms in that list need a plain definition. An electronic health record (EHR) is the hospital’s database for one patient: age, lab results, medicines, vital signs charted at each hour. A time series is a list of measurements taken one after another, such as a heart rate every minute.
The next slide makes a human point. People have five senses: touch, sight, hearing, smell and taste. The slide says that multi-modality is a fundamental characteristic of human life, and it asks: why not for machines?
The syllabus pairs this session with two readings. One is Matthew Turk’s review Multimodal interaction: A review (Pattern Recognition Letters, 2014). It surveys how people and computers use several channels at once, such as speech, gesture and gaze. Read it for the wider map of channels and how they work together. The other reading, Oviatt’s Ten Myths of Multimodal Interaction, comes in Chapter 4, where it helps us judge early and late fusion.
Here is one imagined patient with six findings. Each finding is visible to at least one source. Switch sources on and off, and count how many findings you can still see.
Each row is one finding. The coloured dots show which sources can see it. A green row means a source you switched on can see it. Switch sources and watch the count.
Illustrative: the six findings and which source sees each one are made up for this lesson. The first finding uses the value from the lecture’s note example (Troponin T 0.13 ng/mL, flagged as outside normal).
Notice two things. First, no single source sees everything: the best one sees three of the six findings. Second, some findings show up in more than one source. The note repeats the lab value, so those sources overlap. Real patient data has both properties, and fusion has to handle both.
Before we fuse anything, we need the plain meaning of a prediction model. It is a function with numbers going in and one answer coming out. You show it many past patients, each with a known outcome. It adjusts itself until its answers match those outcomes. Then you give it a new patient and it answers.
The outcome it learns to match is the label. For this lecture the labels are things like “this patient died or went to hospice” or “this patient left the hospital within 48 hours”. The numbers going in are the features. So the whole difficulty of multimodal work sits in one question: how do you turn a picture, a paragraph and a table into one list of features?
A single source can fail the model in two different ways. It can be silent: the finding is simply not in that source, as with the X-ray and the heart rate above. Or it can be noisy: the finding is there, but buried in clutter. Fusion helps with the first problem by adding sources that can see what the others cannot. It helps with the second by letting independent sources vote.
The lecture’s answer has a name: HAIM, Holistic Artificial Intelligence for Medicine. The slide cites the paper Integrated multimodal artificial intelligence framework for healthcare applications, by Luis R. Soenksen, Dimitris Bertsimas and eight co-authors. It appeared in September 2022, and the slide names the journal as Nature Digital Medicine.
The next slide states the vision as a question. The slide opens with the story of IBM Watson, a famous earlier attempt to bring AI to medicine. It then asks: can a holistic perspective on AI, joining computer vision (programs that read images), natural language processing (programs that read text) and machine learning (programs that learn rules from examples), improve how well models predict and prescribe in medicine?
The lecture has four parts, and so does this lesson: the motivation you just read, the HAIM method, HAIM in action at the Hartford HealthCare system, and other areas where the same idea applies. We add the two fusion ideas from the syllabus line, early and late fusion and explainable fusion, at the places where the method needs them. A short note on logistics: this is Week 3.2 of the course, held on 19 February 2026, and the slides are linked in the credit at the top and in the Field Guide.
The toggles in the hero at the top are the four sources, and the next chapters build every part of it. The lecture draws the same picture on its seventh slide: many sources go in on the left, and one unified patient representation comes out on the right.
Chapter 1
Turn a hospital’s table of measurements and a heart-rate trace into a short list of numbers a model can read
A nurse checks a patient every hour and writes down the heart rate. After two days the chart holds 48 readings. A different patient stayed one night and has 12. Both charts are real records, and both are useful. But they do not have the same length.
That is the first problem of fusion. The lecture’s seventh slide poses it as a question: how do we combine different modalities of data? Its picture shows a pile of forms, a chest X-ray, an ECG strip and a doctor’s handwriting, with an arrow into a box. The box is HAIM, and what comes out is a unified patient representation: one object that stands for the whole patient.
A prediction model accepts one thing: a vector. A vector is a list of numbers with a fixed length, for example five numbers. We write its size as a shape. The shape (5,) means “five numbers in a row”. Every patient must give the model a vector of the same shape. If one patient gives five numbers and another gives forty, the model cannot compare them.
So each source needs a converter that takes whatever the source holds and returns a vector of a fixed shape. In machine learning such a converter is often called an encoder, and its output is an embedding. Chapter 2 builds the encoders for text and images. This chapter builds the two simplest ones: for a table and for a time series.
The eighth slide shows a sample of the tabular EHR data. Each row is one event in the patient’s stay. A row has a time it was charted, an item number, a value and a unit, a label such as Heart Rate or Respiratory Rate, a category, and a warning flag. The sample rows include a heart rate of 57 bpm (beats per minute), an alarm labelled “HR Alarm - High” at 120, a respiratory rate of 19 insp/min (breaths per minute) and an oxygen saturation of 98%.
This is a long table: one row per measurement. A patient with a long stay has thousands of rows. A short stay has few. You cannot paste rows into a model, because the number of rows differs from patient to patient. You first need a fixed set of columns, a wide table, with one row per patient and one column per fact, such as age or the last lab value.
Some rows in that table repeat the same measurement over time. Those make a time series. The ninth slide shows the HAIM answer for a time series. It draws one heart-rate trace and marks five things on it: the max, the min, the median, the mean and the number of peaks. Five numbers per series, whatever its length.
Define each one. The max is the largest reading and the min is the smallest. The mean is the sum divided by the count. The median is the middle reading after you sort them. A peak is a reading that is higher than the readings on both sides.
Work one small example by hand. A patient has ten heart-rate readings: 13, 16, 14, 14.5, 12, 13, 15, 14, 12.5, 13. The max is 16 and the min is 12. The sum is 137, so the mean is 13.7. Sorted, the readings are 12, 12.5, 13, 13, 13, 14, 14, 14.5, 15, 16. The fifth is 13 and the sixth is 14, so the median is 13.5. The peaks are 16 (between 13 and 14), 14.5 (between 14 and 12) and 15 (between 13 and 14). That is 3 peaks. The ten readings became the vector (16, 12, 13.5, 13.7, 3), shape (5,).
Now try it on traces of different length. Pick a patient, then change what counts as a peak.
The purple line is the heart rate. Dots mark peaks. Choose a patient with a short or long stay. Then drag the peak threshold from 0 up and watch the peak count fall.
Illustrative: the three traces are generated from a seeded random walk, not patient data. A reading counts as a peak only if it is higher than both neighbours by at least the threshold.
Look at what the threshold does. With a threshold of 0, every tiny wobble is a peak, and the count is large and unstable. With a threshold of 1 bpm, only real rises count. The three patients now give five numbers each, although their traces have 30, 48 and 72 readings.
pythonimport numpy as np def summarize(x, thr=1.0): # x: shape (T,), T differs per patient mid, left, right = x[1:-1], x[:-2], x[2:] peaks = ((mid > left + thr) & (mid > right + thr)).sum() return np.array([x.max(), x.min(), np.median(x), x.mean(), peaks]) # shape (5,) hr = np.array([13, 16, 14, 14.5, 12, 13, 15, 14, 12.5, 13]) print(summarize(hr, thr=0)) # [16. 12. 13.5 13.7 3. ]
If a patient has ten different series (heart rate, breathing rate, oxygen and so on), you run summarize on each and join the results. Ten series give a vector of shape (50,). The shape depends on how many series you chose, never on how long the stay was.
A summary is a trade. It makes every patient fit, and it loses information. A heart rate that spikes on day one and a heart rate that spikes on day three can give the same five numbers. The order of events is gone.
The failure shows up as a symptom, a measurement and a fix. The symptom: two patients with near-identical charts get peak counts of 3 and 14. The measurement: the trace has sensor noise of a few tenths of a bpm, and the threshold was 0, so every ripple counted. The fix: set a threshold above the noise, or smooth the trace first. Use the slider above to see the count settle.
Chapter 2
Turn a nurse’s note, an ECG report and a stack of chest X-rays into vectors of 768 numbers each
A nurse writes: Troponin T: 0.13 ng/mL, Warning: outside normal. A doctor reads that line and thinks of the heart. A model sees only characters. Before it can use the line, the line must become numbers that keep its meaning.
The trick behind this is the embedding. An embedding is a vector placed so that similar things sit close together. Think of a map of a city. Two cafes on the same street are close on the map, and a cafe and an airport are far apart. An embedding does the same for notes and pictures: two notes about the heart land near each other, and a note about the kidneys lands elsewhere. The map just has 768 directions instead of two.
The tenth slide shows the HAIM recipe for notes. The patient’s chart holds a list of event strings with the day each was charted: Education Readiness/Motivation: High (day 1), Troponin T: 0.13 ng/mL, Warning: outside normal (day 1), Respiratory Rate: 23 insp/min (day 2) and Platelet Count: 217 K/uL (day 3). The recipe joins the strings into one block of text, a patient summary, and passes it to a language model. Out comes an embedding, a vector with 768 elements.
The slide title names ClinicalBERT, and the output box calls the result a BioBERT embedding. Both are BERT models: neural networks, built from the transformer you may know from language models, that read a whole text at once and return a vector for each token. The “Clinical” and “Bio” versions were trained on medical text, so they know that troponin belongs near the heart. The number 768 is the width of the base-size BERT, the length of the vector it uses for each token.
The eleventh slide does the same for the ECG. The lecture does not feed the ECG trace itself to the text model. It feeds the cardiologist’s written report: Regular supraventricular tachycardia, most likely sinus tachycardia. Non-specific intraventricular conduction delay… The result is again a 768-element embedding. A report that says “tachycardia” (a fast heartbeat) is a good proxy for the trace.
You can see the idea in a toy. The box below turns your text into six numbers, one for each topic, by counting topic words. It then compares your note with three reference notes using cosine similarity: the cosine of the angle between two vectors. It is 1 when they point the same way and 0 when they share nothing.
Type a note, or tap an example. The pink bars are your note’s six numbers. The right bars say how close it is to each reference note.
Toy: six hand-picked topics (heart, lungs, blood, kidney, wellness, alarm words) and a short word list for each, not a trained model. Real HAIM embeddings have 768 learned directions. The first reference note is the lecture’s patient summary, the second shortens its ECG report, the third is made up.
Try “Sinus tachycardia” against the others. It lands near the ECG reference and far from the blood-count note. That is all an embedding promises: closeness stands for similar meaning. The real model learns its 768 directions from large amounts of text, so you cannot name them the way we named six.
The twelfth slide handles chest X-rays. Several X-ray images of one patient each go through a neural network. The slide does not name its type, but for pictures the usual choice is a convolutional network, which slides small filters over an image to find edges, then shapes, then whole structures. At the end comes a single embedding, again with 768 elements. The slide does not say how the several results are merged into one. Averaging is the usual choice, and it keeps the shape fixed whether the patient had one X-ray or ten.
See the data flow with its shapes. Every arrow keeps a fixed output shape, whatever came in.
python# text: one patient summary, any length -> (768,) ids = tokenizer(summary, truncation=True, max_length=512) # (n_tokens,) integers hidden = bert(ids) # (n_tokens, 768) text_emb = hidden.mean(axis=0) # (768,) one common pooling # images: k X-rays of one patient, each (H, W) -> (768,) per_image = [cnn(img) for img in xrays] # k vectors, each (768,) img_emb = np.mean(per_image, axis=0) # (768,)
Why these choices? The encoder is usually frozen (its weights do not change) because it was already trained on huge amounts of text or images. Training a new one on a hospital’s few thousand patients would waste what the big model learned. Mean pooling is used because it ignores order and count, so a long and a short note both give a vector of the same shape.
BERT-family models read a limited number of tokens at a time, commonly 512, which is a few hundred words. A patient who stays three weeks has a summary far longer than that. The symptom: the embedding of a long stay barely changes between day 10 and day 20. The measurement: print the token count and compare it with the model limit. The fix: split the summary into chunks, embed each, and average them. The lecture does not state this fix; it is the standard remedy.
Chapter 3
Join the embeddings of every source into a single vector first, then let one model read it, and see what to do when a source is missing
You now hold five pieces for one patient: a vector of table facts, a vector of signal summaries, an embedding of the notes, an embedding of the ECG report and an embedding of the X-rays. A doctor would lay the pieces side by side on a desk. A model needs them in one row.
The thirteenth slide is the “Unified HAIM framework”. It draws each source’s embedding next to the others, with all of them feeding one representation of the patient at the top. The step that does the joining is the simplest one in this lesson. You put the vectors end to end.
Joining vectors end to end is concatenation. The fusion type has a name that depends on when the sources meet. In early fusion the sources are joined at the input, before any decision is made, and one model learns from the joined vector. In late fusion each source gets its own model first, and only the answers are combined. Chapter 4 compares the two. HAIM, as the lecture presents it, is early fusion of embeddings.
Work the length. Suppose the table gives 20 numbers and ten signals give 5 each, so 50. Notes, the ECG report and the X-rays give 768 each, so 2,304. The patient vector has 20 + 50 + 768 + 768 + 768 = 2,374 elements, shape (2374,). These sizes for the table and signals are our illustration. Only the three 768s come from the lecture.
The slides show one predictor above the unified vector and do not name it, so our sketch uses the simplest one. Logistic regression multiplies each element of x by a weight, adds them up with a bias b, and squeezes the sum into a probability. A tiny worked case with three elements: x = (1.2, −0.5, 2.0), weights w = (0.8, 1.1, 0.3), b = −1. The sum is 0.8×1.2 + 1.1×(−0.5) + 0.3×2.0 − 1 = 0.96 − 0.55 + 0.6 − 1 = 0.01. The probability is 1/(1 + e−0.01) = 0.5025, about 50%. The hero at the top of the page does this with weights fixed at 1: it adds the sources.
Real patients do not arrive complete. Many have no chest X-ray. Some have no notes yet. But the model expects 2,374 numbers. If the X-ray block is empty, what do you put there?
There are three common answers. Zeros: fill the block with 0. The mean vector: fill with the average of that block over all patients who have it. The mean plus a flag: do that, and also add one extra element that says “this source was missing”. Use the device to see how much of the vector becomes filler.
Each block is one source. Block widths use the square root of their length so the small ones stay visible. A dashed block is missing and has been filled. Switch sources off, and pick how to fill them.
The three 768s come from the lecture’s slides; the table (20) and signals (50) sizes are illustrative. The flag adds one element per source, so the length grows from 2,374 only in the third mode.
Switch off the X-rays, the notes and the ECG report together. Only 70 of the 2,374 elements are real, about 3%. The other 97% is filler. A model trained mostly on complete patients may then act strangely on this one. The flag helps because it lets the model learn a rule such as “when the X-ray flag is on, trust the other blocks more”.
There is a second, quieter trap. The table might hold a heart-rate max of 120, while an embedding element is about 0.03. A model that learns by small steps may let the large numbers dominate. The symptom: the text block seems to do nothing. The measurement: compare the standard deviation of each block. The fix: standardise each column, which means subtract its mean and divide by its spread, so every element is on the same scale.
One more design choice hides in the join: the order of the blocks. For a linear model the order does not matter, because each element has its own weight. What matters is that the order stays the same for every patient, in training and in use. A common bug is to build the vector in one order for the training set and another at the hospital. The model then reads notes as if they were signals, and the scores look random. Keep the block order in one place in the code, and check the shape and a few values before each run.
Chapter 4
Compare joining the sources before the decision with voting after it, break each one on purpose, and read what two classic papers say about when each fits
Two hospitals handle a hard case in two ways. In the first, the radiologist, the lab doctor and the nurse sit at one table. They look at every result together and talk until they agree. In the second, each of them works alone and writes one verdict. A chair then averages the three verdicts.
The first hospital does early fusion: the evidence meets before any verdict exists. The second does late fusion, also called decision-level fusion: each source gets its own model, and only the model outputs are combined. A middle path, hybrid fusion, lets some features meet early and others late. Chapter 3 built early fusion. This chapter tests it against the vote.
Here are the two recipes for two sources, A and B, with n patients. Early fusion makes one model from the joined input. Late fusion makes one model per source and averages their probabilities.
python# early fusion: join first, one model X = np.concatenate([Xa, Xb], axis=1) # (n, da + db) early = Model().fit(X, y) p = early.predict_proba(X)[:, 1] # (n,) # late fusion: one model per source, then vote ma, mb = Model().fit(Xa, y), Model().fit(Xb, y) p = (ma.predict_proba(Xa)[:, 1] + mb.predict_proba(Xb)[:, 1]) / 2 # (n,)
| Question | Early fusion | Late fusion |
|---|---|---|
| Where do sources meet? | At the input, as one vector | At the output, as probabilities |
| Sees how sources interact? | Yes: one model sees both at once | No: each model sees one source |
| A source is missing | Needs a fill rule (Chapter 3) | Skip that model and average the rest |
| Training data | Needs patients with all sources | Each model can use its own patients |
Three toy worlds show where each recipe wins. In the additive world, each source holds a separate clue, and adding the clues works. In the interaction world, a patient is sick when A is high and B is low, or A low and B high. Neither source says anything alone. In the missing world, source B is dropped at test time. Pick a world and a model, then read both accuracies.
Each dot is a test patient. Filled warm means sick, hollow grey means healthy. A red ring marks a wrong call by the model you picked. Change the world, then change the model.
Toy data, seeded: 150 training and 150 test patients per world. Early fusion is a 7-nearest-neighbour vote on the joined (A, B) pair. Late fusion gives each source a simple score from the class means, then adds the scores; when B is missing it adds only A’s score. In the missing world, early fusion fills B with 0, the average.
In the additive world both recipes do well, and they are close. In the interaction world the vote falls to about a coin flip. The reason is exact: each source alone has the same average for sick and healthy patients, so each model learns nothing. Only the joined pair carries the signal. In the missing world, early fusion keeps working a little but has been trained on complete pairs, while the vote simply uses what it has.
Read the failure as symptom, measurement and fix. The symptom: late fusion sits near 50% in the interaction world. The measurement: the accuracy of each single-source model is also near 50%, but a joint model is far above that. The fix: use early or hybrid fusion, so one model sees both sources together.
The syllabus pairs this session with two papers about people and machines that use several channels. Sharon Oviatt’s Ten Myths of Multimodal Interaction (Communications of the ACM, 1999) lists ten popular beliefs about multimodal systems and argues that each is false. Four of them speak straight to fusion design. We state them in our own words from the paper’s well-known list, so check the exact wording in the paper.
The second reading, Matthew Turk’s Multimodal interaction: A review, supplies the wider map of channels such as speech, gesture and gaze. Read it for the vocabulary. For the next session, Week 4.1 on interactions and fusion models goes much further into how to measure which kind of interaction a dataset holds.
Hybrid fusion deserves one more line. A common hybrid trains one small model per source, then feeds their outputs, plus a few raw features, into a second model that learns how to weigh them. It keeps the robustness of the vote, since a missing source can be skipped, and it adds a learned rule for combining. It still cannot see interactions that the first-stage models threw away, so the first stage must keep enough detail.
Chapter 5
Follow how the lecture builds labels from hospital records, tests fusion on three kinds of target, and reports the gain over a single source
Imagine you want to prove that fusion helps. A claim like “it works on our toy cohort” is not enough. You need many real patients, with known outcomes, from a real hospital. And you need a rule that turns each record into a yes or a no.
The fourteenth slide names it: MIMIC-IV. It holds 34,537 patient records from patients in the emergency and intensive-care rooms of Beth Israel Deaconess Medical Center. The slide stresses the “ample sample size”. It lists the modalities in the data: chest X-ray, EHR, radiology notes and ECG. That list matches the five blocks of Chapter 3: table and signals from the EHR, the notes, the ECG report and the X-rays.
One record (also called a sample) is one patient case with all its sources attached. Joining the sources to the right patient and the right time is real work. If a note belongs to a different admission than the X-ray, the fused vector describes nobody.
The fifteenth slide lists what the model must predict. Each target is a yes-or-no question, which makes the task a binary classification. We call the answer the label: 1 for yes, 0 for no.
The labels do not come from a doctor typing “1”. They come from rules over the record. Work that out on a small imagined set of eight patients.
Pick a target. The rule appears at the top, and each patient gets a 1 or a 0. Look at how many patients get a 1.
Illustrative: eight made-up patients to show the rules, not rows from MIMIC-IV. The three rules follow the targets on the lecture’s slide 15; the cut-off for length of stay is the slide’s 48 hours.
The same eight patients give three different yes-counts. Mortality is the rarest of the three. That matters, and we come back to it below.
Here the same rules look like code. The column names are illustrative.
pythonstays["mortality"] = stays.disposition.isin(["DIED", "HOSPICE"]).astype(int) stays["los_48h"] = ((stays.dischtime - stays.admittime) < pd.Timedelta(hours=48)).astype(int) stays["pneumonia"] = labels.merge(stays, on="study_id").pneumonia.fillna(0) # from the report y = stays["mortality"].to_numpy() # shape (n,), values 0 or 1
The sixteenth slide is titled “HAIM significantly outperforms single modality”. It gives one range for each goal.
| Goal | HAIM improvement over a single source |
|---|---|
| Mortality prediction | 11 to 33% |
| Disease classification | 6 to 22% |
| Length of stay | 8 to 20% |
Read these carefully. They are ranges, so they depend on something: probably which single source is the baseline and which disease. The slide does not say what the percentages measure, so we do not guess a metric. What the slide does say is clear: on all three kinds of target, joining the sources beat using one source. That is the same pattern as the hero, in real data.
The first trap is class imbalance. Suppose 8% of patients die. A model that answers “alive” for everyone is right 92% of the time and helps nobody. The symptom is a high accuracy score. The measurement is the share of positives: if it is 8%, then 92% is the floor. The fix is a score that ignores the share, such as AUC (Chapter 7), together with the base rate.
The second trap is leakage: a feature that contains the answer. A note written after death may say “expired”. A model that reads it looks perfect and is useless, because at the moment you need the prediction the note does not exist. The symptom is a near-perfect score. The measurement is to remove the last day of notes and watch the score fall. The fix is to build each patient’s features only from what was known at prediction time.
One more habit makes such a test honest. Before training, set aside a share of the patients as a test set and never touch them while you build. A model can memorise the patients it trains on, so its score on those patients says little. Its score on the held-out ones says how it will behave on a new patient. The lecture’s slides do not give the split, so we describe the habit, not their numbers. When a score looks too good, ask first whether the test patients stayed truly unseen, and whether one patient appears in both sets under two admissions.
Chapter 6
Split one prediction fairly among the sources that produced it, and learn what a Shapley explanation can and cannot tell a doctor
A model says that this patient has a 62% chance of dying in the hospital. The doctor asks the obvious question: why? A single number gives her nothing to check. She wants to know whether the model leaned on the X-ray, the notes, the heart-rate trace or the lab table, so she can compare that with what she sees herself.
The seventeenth slide is titled “SHAP Analysis”. It shows one small chart for each target, and each chart has four rows: vision, text, time series and tabular. A bar on a row shows how far that source pushed the prediction up or down. This is explainable fusion: after joining the sources, say how much each one counted.
Picture four friends who build a sandcastle. The castle is worth more than the sum of what each could build alone, because they help each other. How do you share credit fairly? The economist Lloyd Shapley gave an answer in 1953. Ask each friend: how much did the castle improve when you joined? That depends on who was already there, so average the answer over every order in which the friends could arrive.
For fusion, the friends are the sources. We first need a way to say what the model predicts when only some sources are known. Call that value v(S), for a set S of sources. The sources outside S are replaced by a baseline: typically the average patient’s values. Then v(∅) is the prediction with nothing known, and v(all) is the real prediction.
Work it with two sources first. With nothing known the model says 0.15. With only the X-ray it says 0.50. With only the notes it says 0.35. With both it says 0.80. In the order X-ray then notes, the X-ray adds 0.50 − 0.15 = 0.35 and the notes add 0.80 − 0.50 = 0.30. In the order notes then X-ray, the notes add 0.35 − 0.15 = 0.20 and the X-ray adds 0.80 − 0.35 = 0.45. Average each: X-ray (0.35 + 0.45)/2 = 0.40, notes (0.30 + 0.20)/2 = 0.25. They sum to 0.65, exactly the climb from 0.15 to 0.80. Nothing is lost and nothing is invented.
The device below works the same sum for four sources. Press Next ordering to add the sources one by one in one random order and see each marginal contribution. After 24 orderings the bars are exact.
Choose a patient. Then step through orderings. The bar is the running average credit for each source. The small tick is its contribution in the latest ordering.
Toy: the 16 coalition predictions come from a small made-up logistic score with a baseline of 0.15, one weight per source and, for patient 1, one extra bonus when vision and text are both known. They are not from the lecture’s patients. The four row names follow slide 17.
Watch patient 1. Vision and text get more than their solo weights suggest, because they lift each other. The split shares that bonus between them. For patient 2 the credit goes to time series and tabular, and vision gets a small negative share: it pushed the prediction down.
A real fused vector has 2,374 elements, not four. Exact Shapley over 2,374 players means 2,374! orderings, which no computer can run. Libraries use sampling and a model-specific shortcut instead. HAIM-style explanation then sums the credit of all elements that belong to one source. This grouping is what turns thousands of numbers into the four rows of the slide.
pythonimport shap f = lambda X: model.predict_proba(X)[:, 1] # (n, d) -> (n,) explainer = shap.Explainer(f, background) # background: (100, d) typical patients sv = explainer(x) # sv.values: (1, d), one credit per element blocks = {"tabular": slice(0, 20), "signals": slice(20, 70), "text": slice(70, 1606), "vision": slice(1606, 2374)} credit = {k: sv.values[0, s].sum() for k, s in blocks.items()} # 4 numbers
Two failures are common. First, the credit depends on the baseline, which defines what “absent” means. The symptom: the explanation says text dominates for every patient. The measurement: rerun it with a different background set and see whether the ranking holds. The fix: state the baseline next to the chart.
Second, sources that repeat each other share the credit. If the note copies the lab value, the table and the text each get half. Neither looks important alone. Add them and you see the real weight.
You can also run Shapley the other way, as a check. Remove one source at a time and see how much the prediction falls. That is a cruder tool, because it ignores the orders in which sources arrive. It misses help that two sources give each other. Patient 1 shows it: removing vision alone costs the whole shared bonus, and so does removing text alone, so the two drops count that bonus twice and add up to more than the climb. Averaging over all orderings splits the bonus once, at the price of more model calls.
Chapter 7
Read AUC across seven hospitals, learn what it measures, and check whether a model’s probabilities can be taken at face value
A research score is one thing. A tool that nurses open every morning is another. The second half of the lecture moves from the public dataset to a working hospital system, and the questions change. It is no longer “does fusion beat one source?” It is “can people rely on this, in every building, every day?”
The eighteenth slide introduces Hartford HealthCare: the largest hospital system in Connecticut, with $5 billion in operating revenue and seven hospitals that range from academic to community and from large to small, about 2,500 beds in total. The slide calls it representative of typical US hospital networks. That is the point of the choice: a result here is likely to carry over elsewhere.
The twentieth slide states the problem. Build models that predict patient operational characteristics for every patient hospitalized in the system: discharge in the next 24 and 48 hours, final destination, mortality risk, and ICU risk in the next 24 and 48 hours. The users are physicians, nurses, case managers and bed managers. The uses are to identify and prioritise patients, to reduce length of stay and cost, to prepare treatment and disposition plans, to warn of possible deterioration, and to allocate hospital resources.
The nineteenth slide gives the outcome so far. It describes end-to-end software deployed in production, used by hundreds of people, that assists operational decisions daily at seven hospitals. The slide projects a yearly revenue uplift in the millions of dollars and says the work is scaling to other hospital systems worldwide.
The slide judges each model with AUC, short for area under the ROC curve. You can skip the curve and use the plain meaning: pick one patient who had the event and one who did not. AUC is the chance that the model gave the first patient the higher score. 0.5 means a coin flip and 1.0 means every pair is right.
Count it by hand. Three patients had the event, with scores 0.9, 0.6 and 0.4. Three did not, with scores 0.5, 0.3 and 0.2. That makes 3 × 3 = 9 pairs. The 0.9 patient beats all three. The 0.6 patient beats all three. The 0.4 patient beats 0.3 and 0.2 but loses to 0.5. So 3 + 3 + 2 = 8 pairs are right, and AUC = 8/9 = 0.889. Notice what AUC ignores: how common the event is, and the size of the scores. It only checks the ranking.
Slide 21 is a table of AUCs for all seven hospitals. Explore it.
Pick a target. Each bar is one hospital’s AUC, drawn from 0.5, a coin flip. A hospital without bars has no such unit.
All numbers are from slide 21. The seven column codes (HH, CH, BH, HOCC, MMC, SV, WH) are the slide’s own; the slide does not expand them. The ICU rows have no value for WH, marked “No ICU unit” on the slide.
Look at mortality: every hospital is between 0.888 and 0.925. The seven hospitals range from academic to community, and the scores stay close. The lowest scores sit in WH for discharge, at 0.768 for one day and 0.757 for two days. Discharge is harder than mortality, and the WH site is harder still. The table says that plainly, without rounding it away.
AUC checks the ranking only. A nurse acts on a number such as “70% chance of discharge tomorrow”, so the number itself must be honest. A model is calibrated if, among all patients it gives 70%, about 70 in 100 really have the event. The twenty-second slide plots this for 48-hour discharge and for mortality: predicted probability on one axis, observed frequency on the other. The dots hug the diagonal, and the slide states it plainly: predicted probabilities closely match empirical evidence.
The diagonal is perfect calibration. Drag the slider above 1 to make the model overconfident, or below 1 to make it timid. Watch the dots leave the line.
Illustrative: the observed rate in each bin is the predicted probability with its log-odds divided by the slider value, so 1.00 is perfectly calibrated. This is not the lecture’s data; its two curves are close to the diagonal.
The point of the device: a model can keep the same ranking, and the same AUC, while its probabilities lie. At 2.0 the model says 90% but only about 75% of those patients have the event. An alert rule that fires at 0.5 or 0.35 (Chapter 8) would then fire for the wrong people. Check both numbers, and recalibrate on held-out patients if they disagree.
pythonfrom sklearn.metrics import roc_auc_score from sklearn.calibration import calibration_curve auc = roc_auc_score(y_true, p) # y_true (n,), p (n,) -> one number frac_pos, mean_pred = calibration_curve(y_true, p, n_bins=10) # two arrays of shape (10,) gap = np.abs(frac_pos - mean_pred).mean() # 0 means a perfect diagonal
Why report seven hospitals and not one pooled number? A pooled score can hide a weak site. The lowest discharge scores sit at one hospital, and a nurse there would never see that in an average. When a tool serves many buildings, the lecture’s table is the right shape of evidence: one row per question, one column per site, with every number in view.
Chapter 8
Turn a probability into a green or yellow flag a nurse can act on, compare the tool with doctors’ own estimates, and read the claim about days saved with care
A bed manager starts the morning with forty patients on a screen. She cannot read forty charts. She needs the five who might go home today, and she needs them first. A probability in a database does not help her. A coloured flag in the right row does.
Slides 23 and 24 show the tool. It is a table with one row per patient. Next to each name sit the model’s probabilities and a coloured alert cell. Beside the example patient, titled “from yellow to green”, the tool lists the factors that raised or lowered the probability. That list uses the idea of Chapter 6, applied to the individual features that fed the model.
The slide gives the alert rules in words. A green alert fires if the discharge probability for the next 24 hours or the next 48 hours is over 0.5. A yellow alert fires if the 48-hour discharge probability is between 0.35 and 0.5 and has risen by more than 0.1 since yesterday. Yellow means “getting close”, and green means “plan for discharge now”.
Work one day by hand. Yesterday a patient’s 48-hour probability was 0.30. Today it is 0.41, and the 24-hour probability is 0.14. Is she in the 0.35 to 0.5 band? Yes. Did it rise by more than 0.1? 0.41 − 0.30 = 0.11, so yes. Neither probability is over 0.5. So she gets yellow. If tomorrow her 48-hour probability is 0.58, the first rule fires and she turns green.
The line is a patient’s 48-hour discharge probability for six days. Dots show the alert that day. Pick a patient, then move the green threshold and watch the flags change.
Illustrative: the three six-day probability paths are made up. The rules and their 0.5, 0.35 to 0.5 and 0.1 values are from slide 24; the slider changes only the green threshold. The 24-hour probability is taken as 0.4 of the 48-hour value.
Move the threshold down to 0.35. More days turn green, and the nurse sees more alerts. Move it up to 0.65. Fewer alerts fire, and some patients who were ready go unflagged. There is no free setting. This is the trade between catching more discharges and flagging too many patients, which the next section names.
Slide 25 compares the tool with the doctors’ own estimates. The slide writes EDDs, which we read as estimated discharge dates. It reports the tool’s gain over the EDDs at four hospitals, validated on all inpatients from April to October 2021.
| Metric | HH | CH | BH | HOCC |
|---|---|---|---|---|
| AUC | +0.186 | +0.198 | +0.180 | +0.209 |
| Proportion of discharges detected | +3% | +6% | +1% | +16% |
| Precision of predicted discharges | +10% | +11% | +12% | +12% |
Two new words. Proportion detected asks: of the patients who really left, how many did the tool flag beforehand? Precision asks: of the patients the tool flagged, how many really left? Suppose it flags 100 patients and 70 leave. Precision is 70%. If 140 patients really left, it detected 70 of 140, which is 50%. A tool can raise one by sacrificing the other, which is why the slide reports both. The slide’s summary is that the tool has higher AUCs, identifies more patients who can be discharged, and makes more accurate predictions.
The failure to watch is alert fatigue. The symptom: nurses stop looking at the colours. The measurement: the share of alerts that led to action, which is the same as precision. The fix: raise the threshold, rank the list, or show fewer alerts with better reasons next to them.
Slide 26 reports the effect on length of stay. It calls the analysis preliminary and descriptive. The finding: average patient length of stay fell by 0.67 days. The comparison is between 12 hospital units without full deployment, which act as the control, and 15 units with full deployment after 15 January 2023, measured as of 15 April 2023, on general-care medicine and cardiology units. The slide says it assumes a parallel trend and that more rigorous analysis is ongoing.
The idea behind “parallel trend” is simple. Stays change over time for reasons unrelated to the tool. So compare each group with itself before and after, and then compare the two changes. Imagine (our numbers) that control stays rose by 0.20 days and deployed stays fell by 0.47 days. The effect is the gap, 0.67 days. The method is valid only if the deployed units would have moved like the control units without the tool. The slide is honest that this is still being checked.
The last slide lists current work: detecting domestic abuse at Brigham and Women’s Hospital, an AI companion, automatic detection of data from notes, scans and labs at MGB, unified diagnostics from EKGs, and detection of subdural hematoma (bleeding between the brain and its covering). Each has the same shape as Chapters 1 to 6: several sources, one vector, one prediction, one explanation.
Chapter 9
Keep the numbers, the terms, a checklist, the core code and the ways onward, and pick a fusion recipe for your own problem
The whole lecture fits in one sentence. Turn every source into a vector of fixed length, join the vectors, let one model read the result, check its ranking and its probabilities, explain it by source, and measure what it changes in a real hospital. Early fusion joins the sources before the decision. Late fusion joins the decisions. HAIM, as the lecture presents it, is early fusion of embeddings with Shapley explanations by source.
Turn on the facts that are true for your data. The device scores the three recipes. It is a rule of thumb from this lesson’s toys and tables, not a law.
Each fact adds points to the recipes it favours. Switch facts on and off and watch the bars.
Illustrative scoring made for this lesson. Facts that favour early fusion: sources interact (Chapter 4) and one model gives clean credit by source (Chapter 6). Facts that favour late fusion: missing sources and separate data for each source (Chapters 3 and 4). Hybrid earns a point for every fact that pulls both ways.
| What | Value | Where from |
|---|---|---|
| Length of each text or image embedding | 768 elements | Slides 10 to 12 |
| Time-series summary | max, min, median, mean, number of peaks | Slide 9 |
| Patient records in MIMIC-IV | 34,537 | Slide 14 |
| Disease targets | 10, plus mortality and a 48-hour stay | Slide 15 |
| Gain over one source | 11 to 33% (mortality), 6 to 22% (disease), 8 to 20% (stay) | Slide 16 |
| Hospitals, beds | 7, about 2,500 beds | Slide 18 |
| Mortality AUC across the seven hospitals | 0.888 to 0.925 | Slide 21 |
| Green alert | 24 h or 48 h discharge probability over 0.5 | Slide 24 |
| Yellow alert | 48 h probability 0.35 to 0.5 and up more than 0.1 | Slide 24 |
| AUC gain over doctors’ estimates | +0.180 to +0.209 | Slide 25 |
| Average stay reduction (preliminary) | 0.67 days | Slide 26 |
pythondef patient_vector(p, enc, mean_blocks): blocks = {} blocks["tab"] = p.table # (20,) one row per patient blocks["sig"] = np.concatenate([summarize(s) for s in p.series]) # (5 * n_series,) blocks["txt"] = enc.text(p.notes) if p.notes else None # (768,) blocks["ecg"] = enc.text(p.ecg_report) if p.ecg_report else None # (768,) blocks["img"] = enc.image(p.xrays) if p.xrays else None # (768,) flags = np.array([0 if v is None else 1 for v in blocks.values()]) # (5,) filled = [mean_blocks[k] if v is None else v for k, v in blocks.items()] return np.concatenate(filled + [flags]) # (d + 5,) X = np.stack([patient_vector(p, enc, means) for p in train]) # (n, d + 5), then standardise columns model = LogisticRegression(max_iter=1000).fit(X, y) # y: (n,) mortality labels p_new = model.predict_proba(patient_vector(new, enc, means)[None])[:, 1] # (1,)
This session opens the fusion theme. It gives the vocabulary, one worked application and the idea of explaining by source. The next session asks the deeper question the interaction toy of Chapter 4 only hints at: what kinds of interaction can exist between sources, and how do you measure them? Later sessions cover alignment between modalities and large multimodal models.
This lesson follows Week 3.2 of MIT MMAI: Multimodal AI (Spring 2026), taught by Paul Liang of the MIT Multisensory Intelligence group. The slides, “HAIM: Holistic AI for Medicine” by Dimitris Bertsimas, are linked as the lecture slides. We re-teach every idea in our own words and devices, and we embed none of the course’s images. The slides cite the HAIM paper by Soenksen and colleagues (September 2022). The readings are Ten Myths of Multimodal Interaction (Oviatt, 1999) and Multimodal interaction: A review (Turk, 2014). When this lesson was written the publishers’ pages for both readings would not open, so we describe them only by title and well-known scope. Read them from the links. All toy numbers say so in their fine print.
Now press Present or Teach and explain the pipeline from memory: how a table, a trace, a note and a scan become vectors, how they join, why a vote fails on interactions, how Shapley splits the credit, and what AUC and calibration each check. Then go back to the four sources at the top, switch them on one by one, and say what each one adds.