OpenSLA

A 60-year-old man lies on a bed in an emergency room. He came in with belly pain and nausea. Three small stickers on his chest send his heart's electrical activity to a monitor above the bed. A nurse charts his breathing rate again and again: 31 times in half an hour. He rates his pain a 5.

The screen fills with lines and numbers. His heart beats about 100 times a minute. His breathing is fast: the middle of his 31 readings is 26 breaths a minute. Then someone has to decide. Do we give him something now, and if so, what?

Hold that moment in mind, because this whole lesson lives in it. A monitor can tell you what is happening. It cannot tell you what to do about it. And today, a separate program has to be built for every decision someone wants help with.

What the monitor sees

Start with names for what is on that screen. A sensor is a device that measures the body over time: stickers for the heart, a clip on a finger, a small patch that reads sugar under the skin. Think of a thermometer that never stops reading.

The stream of numbers a sensor writes is a signal. Drawn as a line, a signal is a trace, like the line a seismograph draws. The heart's electrical signal, picked up by the chest stickers, is the ECG (electrocardiogram). You can think of it as listening to the heart's wiring.

Some signals arrive as a continuous wave, like the ECG. Others are numbers a nurse or a machine writes down from time to time, like the 31 breathing rates and the pain score. Both kinds count as signals in this lesson. Both sit on the same timeline.

No model reads a whole hospital stay at once. It looks at a slice of recent past that ends at the moment of decision. That slice is the window: here, the last 30 minutes of the record. Picture a page torn from a long diary. The window is the page, and the decision is what you write at the bottom of it.

What sensor models already do well

Saying what the signals show is called perception. "The heart beats about 100 times a minute and the breathing is fast" is perception. It is reading the weather, not deciding to take an umbrella.

Machines have become very good at this. A sensor model is a network trained on large amounts of sensor data to recognise states or predict outcomes. There are families of them for heart traces, for the pulse wave a finger clip sees, for glucose, for motion and for sleep. Each one is a very good weather reader.

Now look at what they hand back: a state ("fast heart"), or a risk ("likely to get worse"). None of them says "give an anti-nausea medicine now". The step from what is happening to what to do is left to someone else.

The gap: a decision needs the story

A nurse does not decide from the signals alone. She also reads the context: what the record says about the person. Their age, why they came in, their history, what was already done. It is the story behind the numbers.

Change the story, and the right action can change while every number stays the same. A fast heart and fast breathing mean one thing in a man with nausea. They mean another in someone with a fever and a cough, and something else again after a fall. The device below makes this concrete.

Look at what never moved. The two traces stayed put for all four stories, so a model that reads only the traces receives the same input four times. A program given the same input must return the same output. It says "act, 0.62" every time, while the recorded action changed with each story.

The first story is real. It is the record the paper uses as its main example, and the action recorded next was an anti-nausea medicine. The other two stories and their actions are made up to show the idea. The lesson holds anyway: no answer computed from the signals alone can be right for all three.

How decisions are modelled today

When people do build a program to help with a decision, they usually build a classifier: a model that picks one answer from a fixed list. Think of a vending machine. It can only give you the buttons it has.

That fixed list is a closed label space: a list of answers chosen before training, with no "something else" on the menu. Anything not on the list cannot be answered at all.

Count how long such lists get. The paper's benchmark gathers 60 groups of actions. There are 39 in emergency and intensive care, 17 in the operating room, and 4 ranges of insulin dose for people who wear an insulin pump: 39 + 17 + 4 = 60. A real hospital's list of medicines and procedures is far longer, and it changes every year.

Why is a closed list a problem? Follow the argument in four steps.

1. One output per answer. A classifier with K answers ends in K output numbers, one per answer, and each has its own learned weights.
2. A new answer has nothing. A new action, number K + 1, has no output and no weights. The model cannot even score it until someone collects labelled windows and retrains.
3. Separate models learn alone. With N separate decisions you build N such models. None of them learns from the others' data.
4. Score names instead. If each action is scored by comparing the window with the action's name, a new name is just new text, and it can be scored at once. Chapter 5 builds this. Chapter 11 tests it on a medicine the model never trained on.

One more gap remains, quieter than the others. A classifier gives an answer and nothing else. It cannot say "because the pain score is 5 and he came in with nausea". A nurse who must act on its advice has no way to check its reasons.

Language as the link

Notice what the missing pieces have in common. The patient's story is text. Every action has a name, which is text. An explanation is text. So suppose a model could read the signals as if they were words, sitting next to the story. Then it could answer in text too: the decision, a description of the window, and the reasons.

That is the idea of this paper. It calls it Sensor-Language-Action (SLA) modelling: a model that reads signals and words together and answers in words, actions included. Think of a colleague who reads the chart and the notes, then tells you what to do and why. The paper's model is OpenSLA. To the authors' knowledge, it is the first general framework that models sensor readings, language and actions together.

OpenSLA gives back three things for every window. First, a decision on a decision ladder: three questions in order, act or wait, what kind of action, and which one exactly. Think of a tree of road signs: left or right, then which street, then which house. Second, a description of the window, with its numbers. Third, the evidence behind the decision: the findings that fit it.

Go back to the hero and pick OpenSLA. The ladder is the three bars at the bottom. The description is "What is happening?". The evidence is the yellow glow on the pain score, on the breathing readings and on the word nausea.

Two cautions before we build anything. OpenSLA learns from what people recorded doing, not from what was right, so a habit in the records becomes a habit in the model. And the paper says plainly that OpenSLA has not been clinically validated. Chapter 12 opens both. This lesson explains how the model works and how it scored, never what to do for a patient.

Here is the road, in order. Each chapter explains one part of the instrument in the hero.

1. One question: the thirty minutes and the "now: decide" line, and how a long record becomes labelled windows.
2. Captions: where "What is happening?" comes from, four kinds of training sentences.
3. Tokens: how the traces reach a language model at all.
4. Memory: how thousands of signal tokens shrink to a few hundred.
5. Three heads: the ladder, act, what kind, which one.
6. Decoder: "Why this action?" and the yellow evidence.
7. Scoring: the score bars and the coin line.
8. Results: the three partners, scored on six groups of patients.
9. Read back: the chat model's "about 90", and how to check a caption's numbers.
10. What helps: why writing sentences in training improves the decision.
11. New cases: a medicine never seen in training, and a new hospital.
12. Bedside: the fine print, from a test score to a real ward.
13. Map: every number, the checklist and the code in one place.

Picture three screens. The first is the emergency-room monitor from Chapter 0. The second is an anaesthesia screen in an operating room, where a drug drip is nudged every few minutes during surgery. The third is a phone app that shows a sugar line from a small patch on someone's arm, next to the insulin pump on their belt.

Three settings, three kinds of decision: give a medicine, change a drip, give a dose of insulin. They look like three separate problems. This chapter turns them into one question, asked the same way every time.

Here is the question in plain words. At a chosen moment, given the recent signals and the person's story, does someone act in the next few minutes? And if so, how? Everything the model learns is an answer to that question, so it pays to see exactly how one example of it is built.

Three settings and seven groups of people

The paper gathers seven datasets of real, anonymised records, grouped into three settings. In this lesson each one gets a plain nickname; the real name is in brackets the first time only.

• Emergency and intensive care. The emergency set (MC-MED) and the intensive-care set (MIMIC-III). A third, the new-hospital set (MIMIC-IV), is held back and never trained on. Chapter 11 uses it.
• The operating room. Surgery set A (MOVER) and surgery set B (VitalDB).
• Daily life with diabetes. Pump set A (MetaboNet) and pump set B (PEDAP).

The pump sets need three words of their own. A continuous glucose monitor (CGM) is a patch that reads blood sugar under the skin every five minutes, like a fuel gauge that updates on its own. A bolus is a single dose of insulin given at once, usually around a meal. Basal insulin is the slow, steady drip a pump gives all day. Think of a meal-time top-up next to a pilot light.

One sample: now, the past, and the next few minutes

Every example the model learns from is anchored at one moment. The decision time, written T₀, is the moment the model must answer: its "now". Think of a referee who must blow the whistle or not at one exact instant.

Before T₀ comes the history window from Chapter 0. Its length depends on the setting: 30 minutes in emergency and intensive care, 5 minutes in the operating room, and 2 hours for the pump. For the pump that is 24 readings, one every five minutes (24 × 5 = 120 minutes). Next to the signals sits the context: the person's background, why they came, and earlier actions, written as text.

After T₀ comes the target interval: the few minutes after now in which an action counts as the answer. Think of the short window in which a goal still counts after a pass. In emergency and intensive care it runs from T₀ to five minutes after it. In the operating room it starts just after T₀ and ends two minutes later. For the pump it is the decision slot itself: does a bolus start right now?

Slide the decision time along a toy record and watch the label flip.

The label flips from No to Yes the moment an action enters the warm shading, and back the moment it leaves. That is the whole labelling rule, written as one line:

y(T₀)

The act-or-wait label for the window that ends at T₀: 1 means someone acted, 0 means nobody did.

a

A recorded action time, read from the record's log of actions.

T₀

The decision time: the end of the history window and the start of the target.

Δ

The target length: 5 minutes in emergency and intensive care, 2 in the operating room. Emergency and intensive care also count an action exactly at T₀; the pump uses the decision slot itself.

To build many examples from one long record, repeat four steps.

1. Read the log. Take a record and its list of action times a₁, a₂, and so on.
2. Pick a now. Choose a decision time T₀. The history is everything in the window before it.
3. Label it. The label is 1 if any action time falls inside the target interval, else 0.
4. Slide and sample. Move T₀ along the record to make many windows. Keep the ones with actions, and sample matched ones without.

Three rungs of labels

A Yes is only the first rung of Chapter 0's ladder. Each window also carries two more labels. The necessity label says whether anyone acts. The category label says what kind of action. The fine-grained label says which one exactly. Think of shopping: yes or no, then the aisle, then the item.

The paper gives its own example. Giving the antibiotic Imipenem is "Yes" at the necessity rung, "antibiotics" at the category rung, and "Imipenem" at the fine-grained rung. Remember that drug; Chapter 11 brings it back.

Two settings bend the ladder. In hospital settings, several kinds of action can happen in the same few minutes, so several categories can be right at once. And the pump has no third rung. Its category is a dose bin: a range of doses treated as one answer. There are four bins: under 1 unit, 1 to 2, 2 to 4, and 4 units or more. Think of shoe sizes instead of exact foot length.

How long is each list? The emergency set has 29 categories and 56 fine-grained labels. The intensive-care set has 28 and 60. Surgery set A has 13 and 22, and surgery set B has 5 and 14. Both pump sets have their 4 dose bins and nothing below. Across the three settings that makes the 60 action groups of Chapter 0.

Windows with and without actions

A positive window is one where at least one listed action falls in its target interval. A negative window has none. Think of a goal and a near-miss.

Choosing negatives is harder than it looks. In hospital settings the paper uses history-matched controls: quiet windows picked to look like the action windows' recent past. It is like comparing twins, not strangers. For the pumps, quiet windows exclude moments with a recent bolus. The story at the end of this chapter shows why that rule matters.

Then the data is split three ways: training windows to learn from, validation windows to tune settings on, and test windows to score on once. The split is individual-disjoint: no person appears in two of the three parts. It is a new class of students for the exam, not the same students with new questions. Within each setting, the training windows of its datasets are pooled.

The test windows are balanced: half have an action and half do not. A fixed set of test cases that every model is scored on is called a benchmark, like the same driving-test route for everyone. Chapter 7 shows why a balanced benchmark changes how a score reads.

How big it all is

Here is the scale, worked out from the paper's tables. Every number below is the paper's; the sums and shares are ours.

worked exampleSamples in one channel of one window
  emergency, intensive care   30 min × 60 s × 125 a second   =   225,000
  operating room               5 min × 60 s × 125 a second   =    37,500
  insulin pump                24 readings × 5 minutes        =   120 minutes

Training windows
  emergency + intensive care       90,412 +    16,184        =   106,596
  surgery A + surgery B           113,865 +    48,800        =   162,665
  pump A + pump B               1,945,570 +   127,437        = 2,073,007
  all three settings                                         = 2,342,268
  pumps' share                  2,073,007 ÷ 2,342,268        =      88.5%
  pump set A alone              1,945,570 ÷ 2,342,268        =     83.06%

Test windows (half with an action, half without)
  2,753 + 3,402 + 795 + 2,000 + 10,000 + 10,000              =    28,950
  plus the held-back new-hospital set                        =     1,000

People
  49,196 + 35,276 + 122 + 26,329 + 3,657 + 1,095 + 65        =   115,740

Two things stand out. First, the people: 115,740 of them across seven datasets, which the paper rounds to "more than 116,000". Second, the training windows are lopsided. Glucose readings are cheap and arrive all day, so the pump sets give 88.5% of all training windows. Pump set A alone gives 83.06%.

Here is one sample as the pipeline stores it. The drug name is illustrative, and Chapter 3 gives the exact tensor shapes.

one sample{
  "decision_time": "minute 30",
  "history": {
    "waveform": "[30 minutes x 3 channels x 7,500 samples]",
    "numeric_events": "31 breathing-rate readings, 1 pain score"
  },
  "context": "60-year-old man, belly pain and nausea, ...",
  "target": { "necessity": 1, "categories": ["antiemetic"], "labels": ["ondansetron"] }
}

Read it top to bottom. The decision time is the end of the window. The history holds the wave, cut into thirty one-minute slots of 7,500 samples each, and the charted numbers as a list of events. The context is plain text. The target holds all three rungs: necessity 1, one category, one fine-grained label.

A pump model that learned the pump, not the decision

What the team sees. A team builds its own pump dataset. For quiet windows it picks random moments from each day. On validation the act-or-wait score looks superb.

What they measure. They score again, with every quiet window that sits within an hour after a bolus removed. The score drops hard.

Why. Right after a dose, "no new dose" is easy to guess from the pump's own log. Many random quiet windows sat right after a bolus, so the model learned the pump's history, not the decision. What they change: the paper's rule, quiet windows exclude recent boluses. The story and the hour are illustrative; the rule is the paper's.

Show the hero's thirty minutes to a student nurse and ask her what she sees. A weak answer is "he looks unwell". A good answer has four parts. It has numbers: heart about 100, breathing about 26. It has links: the charted breathing and the breathing wave agree. It has the person: this is a man who came in with nausea. And it has the point: these findings bear on the anti-nausea medicine.

OpenSLA is trained to give the good answer. The training data does not only say which action came next. It also says, in sentences, what the window showed and which findings bear on that action.

Why sentences, and not just labels

Think about how little an action label says. For each window it is one yes or no, plus a kind and a name. A paragraph about the same window carries a dozen numbers and the relations between them. A student asked to show their work learns more than one asked only for the final answer.

That is the bet of this chapter, and it pays off in a measurable way. When the paper trains the same model on the actions alone, with no sentences, its act-or-wait score on surgery set B falls from 74.8 to 54.1. A coin scores 50 on that scale. Chapter 7 explains the score, and Chapter 10 takes the result apart.

Four kinds of sentence

Each training window comes with a caption: a short paragraph that says in words what the window shows. Think of the note a nurse leaves for the next shift. The caption is built from four kinds of sentence, which the paper calls facets. A facet is one section of a report card.

• Observational. What each channel shows, with numbers. From the hero's record: "The lead II ECG showed an irregular-periodicity proxy, with an estimated rate of 100.2 beats/min."
• Relational. How two channels relate. Only predefined pairs that belong together in the body are compared, such as the charted breathing rate and the breathing wave. The pipeline never hunts for any correlation it can find.
• Global. The person's overall state, placed in their context. The pipeline picks it from a fixed set of state descriptions, ranked by how many findings support each. If none is supported well enough, the caption says so.
• Action. Which findings bear on the recorded action. It is phrased as an association: evidence that fits the action, not a claim that it caused it, and never a recommendation. "Umbrellas and rain go together."

Who writes these sentences? Not a language model. The captions are rule-based. A program extracts facts from the window: each fact has a measurement name, its numbers, its timing and its source. Rules select the facts that matter. Then each selected fact fills a template, a sentence with blanks that a program fills, like a form letter.

Here are two of the paper's real templates. The first builds an action sentence; the second builds a glucose sentence for the pump.

templates"Findings that bear on {domain} are {clause}."
"Median sensor glucose over {period} was {median} mg/dL, {band_phrase}; readings {lo}-{hi} mg/dL."

Build a caption yourself. Each chip on top is one extracted fact; each box below is one facet. A line joins a fact to every facet sentence that uses it.

Every number in the caption traces back to a chip. That is the point of building captions by rule: nothing in them is invented, so every number can be checked. Chapter 9 uses exactly this property to score the model's own captions.

Numbers that one typo cannot move

A charted measurement is summarised by three numbers: the median, the range and n. The median is the middle value once the readings are sorted. The range is the lowest to the highest. And n is how many readings there were. Think of a race: the middle runner, the first and last, and the size of the field.

Why the median and not the average? Because charts contain typos. Suppose a nurse types 80 instead of 30 for one breathing rate. The average of all 31 readings moves; the middle one does not. The worked example below does it with the hero's readings.

Numbers read from the wave itself

A wave needs different facts. The pipeline measures a rate proxy: a rate worked out from the shape of the wave itself, not read from a charted number. It is like timing a heartbeat by counting the bumps on the trace. A rate of 100.2 beats a minute means one beat every 60 ÷ 100.2 = 0.599 seconds.

Next to the rate it reports an IQR (interquartile range): the spread of the middle half of a set of values. Think of how tightly the middle of the pack is bunched in a race. A caption from the paper reads "rate proxy 95.2/min, IQR 0.40": a rate, then how much the measurement spreads around it.

Declare, do not describe

Some channels are too noisy to read. Take the pleth (plethysmography): the pulse wave a finger clip sees by shining light through the skin, like watching a pipe swell with each push. If the clip slips or the hand moves, the wave turns to scribble.

Each channel must pass a signal quality threshold: the bar a channel must clear before the caption reports numbers from it. Think of a judge who refuses to time a blurry photo finish. Below the bar, the pipeline does not guess and does not stay silent. It declares the problem: "The Pleth trace did not meet the quality threshold for morphology or rate-proxy statements." Try it in the device: drag the quality below 60.

The pump's caption

The pump captions follow the same rules with different facts. Glucose is read in mg/dL: milligrams of sugar in a tenth of a litre of blood, like grams of sugar per cup. Three thresholds turn numbers into words. The target band is 70 to 180 mg/dL. Glucose is called "flat" when its slope is under 10 mg/dL per hour. A variability flag goes up when the spread of the readings reaches 60 mg/dL.

The pump also logs tiny deliveries that are not real doses, so the pipeline sets a floor. Deliveries under 0.25 units per row are not counted as a bolus. Pick "Insulin pump" in the device to read the paper's Example 6 in full: a 48-year-old man, two hours of zero basal insulin, glucose between 71 and 100, and the action sentence "Findings on the side of no bolus".

Here is every number in this chapter, worked out.

worked exampleThirty-one charted breathing rates (values illustrative; median, range and n are Figure 2's)
  readings   25 26 24 27 26 23 25 28 26 27 24 26 25 29 26 27
             25 26 28 24 26 30 27 25 26 28 27 26 24 29 27
  sorted     23, 24 ×4, 25 ×5, 26 ×9, 27 ×6, 28 ×3, 29 ×2, 30
  median     the 16th of 31 sorted values           = 26
  range      smallest to largest                    = 23 to 30
  n          how many readings                      = 31
  mean       812 ÷ 31                               = 26.19

One typo: the 30 is charted as 80
  mean       862 ÷ 31                               = 27.81   (moved 1.62)
  median     still the 16th value                   = 26      (did not move)

ECG rate proxy of 100.2 a minute
  one beat every 60 ÷ 100.2                         = 0.599 s

Pump, Example 6
  BMI        104 kg ÷ 1.70² m² = 104 ÷ 2.89         = 35.99, printed 36.0
  slope      +6 mg/dL per hour × 2 hours            = +12 mg/dL in total
  flat?      6 is under 10 mg/dL per hour           = "flat"

Pump, Example 10
  basal      0.90 units an hour × 2 hours           = 1.80 units

+6 mg/dL per hour × 2 hours = +12 mg/dL (under 10 an hour, so "flat")

A caption that described a slipped finger clip

What the team sees. A team builds its own caption pipeline, and it writes a pulse rate from the finger clip for every window, noisy or not. Their trained model starts writing confident pulse rates for windows where the clip had slipped off.

What they measure. They compare the model's pulse rates with reference rates, split by the windows flagged as low quality. The error on those windows is far above the error on clean ones. The model learned to write a number whatever the wave looked like, because its training captions always had one.

What they change. Declare instead of describe, which is the paper's rule. A caption that admits "I cannot read this channel" teaches the model to admit it too. The story is illustrative; the rule is the paper's.

One more thing is new here. Earlier sensor-language studies also captioned signals, but to match text with signals, not to explain actions. None of the four the paper compares itself with had action labels or action evidence. The action facet is what turns a description into a reason.

A language model reads one thing: a row of tokens. The sentence "He came in with nausea" becomes a handful of them. Thirty minutes of one heart wave, sampled 125 times a second, is 30 × 60 × 125 = 225,000 numbers. How do those fit in the same row as the words?

Here is the answer in plain words. Each sensor stream is cut into short pieces. A frozen network turns each piece into a list of numbers. That list then sits in the row like one more word.

What a language model reads

A language model is a network trained to predict the next piece of text. Think of autocomplete grown very large. The pieces it reads and writes are tokens: one slot in the row, usually a piece of a word. Think of one bead on a string.

The model never sees letters. Each token is first replaced by a vector (also called an embedding): a list of numbers that stands for it, like a token's coordinates. Layer by layer, each position's vector is updated by looking at the vectors before it. The vector a position holds after the last layer is its hidden state: the reader's understanding at that point on the page.

OpenSLA's language model sits at the centre of everything, so it is called the backbone. It holds 2,048 numbers per position, has 24 layers and knows a vocabulary of 248,320 tokens. It is a model of about 2 billion parameters, the "2B" in its name. A parameter is one adjustable number in a network, one knob on a huge mixing desk.

So the job is now precise. Somehow, each piece of signal must become a vector of 2,048 numbers, placed in the row where a word's vector would go.

The input, exactly as the code expects it

The released code fixes the shape of everything that goes in. A tensor is a block of numbers with named sides, like a spreadsheet with extra dimensions. A batch is several windows processed together, like a tray of samples. Here are the tensors for the waves and the charted numbers:

input contract# waves (emergency, intensive care, operating room)
waveform               [B, M, C, T]   # T = 7500 samples per one-minute slot (60 s × 125 a second)
waveform_channel_mask  [B, M, C]      # True where that channel really recorded in that minute
waveform_modality_ids  [B, C]         # which named channel each column is ("II", "PLETH", "RESP", ...)
# charted numbers, one entry per reading
values, rel_time_min, measure_ids, source_ids, event_mask     # five parallel [B, E] tensors
summary_features                      # per measurement: 8 numbers (version 1) or 12 (version 2)

Read it line by line. B is the number of windows in the batch. M counts the one-minute slots, so 30 for an emergency window. C counts the channels. T is 7,500: sixty seconds at 125 samples a second, the sampling rate (how many readings a sensor takes each second, like frames per second in a film).

The mask is a yes-or-no grid that says which entries are real, like a stencil. Channels drop out all the time, so the mask matters; it gets its own story below. The modality ids tell the model which column is the ECG and which is the finger clip.

Charted numbers arrive differently, as numeric events: one charted reading each, with its value, when it was taken, and what it measures, like one line in a logbook. The five event tensors are parallel lists of length E. One holds the values. One holds the time, counted in minutes before the decision. One says what was measured, one where it came from, and the last says which entries are real.

The model also gets a short summary per measurement. Version 1 has 8 numbers: whether it was observed, how many readings, the last, mean, smallest and largest value, the slope per minute, and the age of the last reading. Version 2 adds the median, the IQR, the first value, and last minus first.

Three encoders, three reasons

An encoder is a network that turns raw input into vectors: a translator into the model's language. OpenSLA uses three, one for each kind of input, and each choice has a reason.

• Waves get a big frozen encoder. Hospital and operating-room waves are long and rich. They go through an encoder pretrained on large amounts of unlabelled signal, adapted from the same group's sleep model. It is frozen: its numbers are not changed in training, like a reference book you read but never edit. Its pretraining already taught it what heart and breathing waves look like.
• Glucose gets a small patchifier. A pump window is only 24 readings, which does not need a big encoder. A patchifier is a small trainable layer that cuts a short trace into pieces and turns each into a vector, like a cookie cutter.
• Charted numbers get a small trainable Transformer. Each event becomes a vector built from three parts added together: its value, which measurement it is, and when it was taken.

Patches: a film strip for signals

The wave encoder does not read 7,500 samples at once. It reads short pieces of one channel called patches, like the frames of a film strip. The paper does not print the patch length, so this chapter assumes one patch per second: 60 per minute, each holding 125 samples. That number is illustrative; the measured token counts below are not.

The base model, OpenSLA-B, does the simplest thing possible. It projects every patch vector to 2,048 numbers and places them all in front of the prompt, in one long flat row. Tokens placed before the prompt's words are a prefix, like a preface before the chapter.

Count them. Thirty minutes, three channels, sixty patches a minute: 30 × 3 × 60 = 5,400 tokens before the first word of the prompt. The paper measured the real averages per window: 5,801.0 sensor tokens in emergency and intensive care, 3,915.8 in the operating room, and 33.0 for the pump.

Why length costs

Long rows are expensive because of attention: each token compares itself with every other token and mixes in what it finds. Think of everyone in a room checking in with everyone else. Chapter 4 builds it step by step.

The trouble is the "everyone with everyone". With n tokens there are n × n pairs. The measured 5,801 sensor tokens make 5,801 × 5,801 = 33.65 million pairs in every layer. 400 tokens would make 160 thousand, about 210 times fewer. Double the window and the pairs do not double; they quadruple. Try it.

Two formulas sum it up. The first counts the tokens; the second counts the pairs attention compares.

n_sensor

The number of sensor tokens in OpenSLA-B's flat prefix.

M

Minutes in the window: 30 in emergency and intensive care, 5 in the operating room.

C

Channels that recorded.

p

Patches per minute (60 here, illustrative).

n_text

The prompt's own tokens. The device and the numbers in this chapter count the sensor tokens alone, which is where almost all of the pairs come from.

worked exampleFlat prefix, one patch a second (illustrative)
  30 min × 3 channels × 60 patches              = 5,400 tokens
  30 min × 4 channels × 60 patches              = 7,200 tokens
   5 min × 4 channels × 60 patches              = 1,200 tokens  (an operating-room window)
Measured averages (Table 21)
  emergency and intensive care                  = 5,801.0
  operating room                                = 3,915.8
  insulin pump                                  =    33.0
Attention pairs per layer
  5,801 × 5,801                                 = 33.65 million
    400.4 × 400.4                               = 160,320
  ratio                                         = 209.9 times fewer

Chapter 4 shows the catch: cutting the tokens 14.49 times made the model only 3.4 times faster, because the rest of the network costs time too.

The shape walk, as code

Here is the whole path from a raw wave to the prefix, in numpy. It runs as is. The encoder is a random stand-in; what matters is the shape at every step.

pythonimport numpy as np
rng = np.random.default_rng(0)
M, C, T = 30, 3, 7500                 # 30 one-minute slots, 3 channels, 7,500 samples a minute (125 a second)
p = 60                                # patches per minute (illustrative: one per second)
D, d = 16, 2048                       # toy encoder width; the language model's width
waveform = rng.normal(size=(1, M, C, T))                           # [B, M, C, T]
mask = np.ones((1, M, C), dtype=bool); mask[0, :10, 1] = False      # the finger clip was off for 10 minutes
patches = waveform.reshape(1, M, C, p, T // p)                      # [1, 30, 3, 60, 125]
W_enc = rng.normal(size=(T // p, D)) / np.sqrt(T // p)              # stand-in for the frozen encoder
tokens = patches @ W_enc                                            # [1, 30, 3, 60, 16]
keep = np.repeat(mask[..., None], p, axis=-1).reshape(1, -1)         # drop patches from minutes that were not recorded
flat = tokens.reshape(1, M * C * p, D)[keep].reshape(1, -1, D)       # [1, n, 16]
W_proj = rng.normal(size=(D, d)) / np.sqrt(D)                        # the trained projection into the model's width
prefix = flat @ W_proj                                              # [1, n, 2048], placed before the prompt's tokens
print(waveform.shape, patches.shape, tokens.shape, flat.shape, prefix.shape)
# (1, 30, 3, 7500) (1, 30, 3, 60, 125) (1, 30, 3, 60, 16) (1, 4800, 16) (1, 4800, 2048)

Follow the shapes. The wave starts as [1, 30, 3, 7500]. Splitting each minute into 60 patches of 125 samples gives [1, 30, 3, 60, 125]. The encoder turns each patch into 16 numbers. Flattening minutes, channels and patches gives one row of 30 × 3 × 60 = 5,400 tokens.

Then the mask removes the 10 minutes the finger clip was off: 10 × 60 = 600 patches. That leaves 5,400 − 600 = 4,800 tokens. The projection lifts each to 2,048 numbers, and the result joins the prompt's own token vectors in front of the language model.

A model that learned that a missing clip means "do nothing"

What the team sees. A team feeds its model a full grid every time. When the finger clip was off, it fills those minutes with zeros. The model starts saying "no action" whenever the clip is off.

What they measure. They split the test windows into "clip present" and "clip absent". The score on absent windows collapses, and the model's captions talk about a flat pulse wave. To the model, a run of zeros looks exactly like a real, flat signal.

What they change. They pass the mask, so absent minutes leave the row instead of entering it as fake readings. In the released code that is the waveform_channel_mask line above. Missing channels are common: waves cover 70.8% of the emergency set's training windows, 78.8% of the intensive-care set's, and only 25.1% of surgery set A's. The story is illustrative; the mask and the coverage are real.

The flat prefix is not the only way in. A model could write the numbers out as text, but the rows get very long; the chat models in Chapter 8 run small Python programs on the signals instead. A small projector could map sensor vectors into a frozen language model. Or new attention layers inside a frozen model could read them. Chapter 10 compares the last two with OpenSLA's choice and finds a large gap.

At the end of a shift, a nurse does not read out every heartbeat to the next nurse. She says something like: "Heart fast from about minute twenty. Breathing fast and steady all along. Overall uncomfortable and nauseous." A few notes, at three zoom levels: a moment, a channel, the whole picture.

OpenSLA's second model does the same thing to its sensor tokens. Instead of passing every patch to the language model, OpenSLA-H asks a fixed set of learned questions of the patches and passes on only the answers. To see how a question can "read" thousands of tokens, we first need attention itself.

Attention, from zero

Picture a library. You walk in with a question. Every book has a catalogue card that says what it is about, and the book itself holds the contents. You compare your question with every card, decide how much each book matters, and leave with a blend of the contents of the books that matched best.

Attention is that library in numbers. The question is a vector called the query. Every token offers a key (its catalogue card) and a value (its contents). All three are vectors of the same width, D.

The match between a query and a key is their dot product: multiply the matching entries and add them up. It is large when the two vectors point the same way, like two arrows that agree. Each match is divided by the square root of the width, √D, so that wide vectors do not produce huge scores.

Then the scores become weights through softmax: exponentiate each score and divide by the total. The weights are all positive and add up to 1. Think of sharing out exactly one unit of attention. The answer is the weighted average of the values.

Attn

One attention step: N queries read n tokens and return N answers.

Q

The queries, N of them, each D numbers wide (N × D).

K

The n input tokens' keys (n × D). Q K^⊤ holds every query's dot product with every key.

V

The n input tokens' values (n × D): what gets blended into the answer.

softmax

Applied along each query's row of scores, so each row of weights adds up to 1.

D

The width of the vectors. Dividing by √D keeps the scores in a sensible range.

Do it once by hand, in two dimensions. The query is q = (1, 0). Three tokens have keys (2, 0), (0, 1) and (1, 1), and values 10, 0 and 4. The width is D = 2, so we divide by √2 = 1.414.

worked exampleOne query reads three tokens (D = 2)
  scores        (1,0)·(2,0) / 1.414 = 2 / 1.414   = 1.414
                (1,0)·(0,1) / 1.414 = 0           = 0
                (1,0)·(1,1) / 1.414 = 1 / 1.414   = 0.707
  exponentials  e^1.414, e^0, e^0.707             = 4.113, 1.000, 2.028   total 7.141
  weights       4.113 ÷ 7.141, 1 ÷ 7.141, ...     = 0.576, 0.140, 0.284   (they add up to 1)
  answer        0.576 × 10 + 0.140 × 0 + 0.284 × 4
                = 5.760 + 0 + 1.136               = 6.896

The query "wanted" the direction of the first key, so the answer leans toward the first value, 10. It still mixes in a little of the others, because softmax never gives exactly zero.

Questions that come from outside

In Chapter 3's attention, every token asked its own question of every other token. In cross-attention the questions come from somewhere else. Think of an interviewer with a fixed list of questions, reading a stack of documents.

OpenSLA-H uses learned queries: question vectors the network learns during training, not computed from the input. They are a standing checklist. Here is the consequence that matters. The output has one vector per query, however many tokens are read. Look at the formula: N queries in, N answers out. The input count n appears only inside the sums.

That is how a memory gets a fixed number of slots. Point 64 learned queries at 5,400 tokens and you get 64 vectors back. Point them at 10,800 and you still get 64.

The Hierarchical Memory

The paper's module is the Hierarchical Memory. It has two pathways. The first applies layers of queries in sequence, each layer with fewer queries than the one before, so each output token sums up a longer stretch. The second pathway's global queries read the original tokens directly, to catch whatever matters about the whole window. The two outputs are joined.

Z⁽⁰⁾

The frozen encoder's tokens: N₀ of them, each D numbers wide.

Z^(l)

Layer l's output: N_l tokens.

Q_l

Layer l's learned queries.

N_l < N_l−1

Each layer has fewer queries than the last, so each token covers more.

Z_g

The global pathway's output, read straight from Z⁽⁰⁾ by the global queries Q_g. The memory is every layer's output and Z_g, joined.

In practice the layers follow the shape of the data. Local queries read the patches inside one channel-minute. Temporal queries read one channel's local tokens across all its minutes. Global queries read everything. Think of notes per hour, per patient chart, and one line for the whole ward.

The released code sets the sizes: 2 local tokens per channel-minute, 16 per channel, and 64 global. Each stage uses 8 attention heads, then a feed-forward block four times wider. Channel embeddings are added at both stages and time embeddings on the local queries, so every token knows where it came from. Waves and charted numbers get separate parameters. These are the code's defaults; the shipped model may override them.

Count the memory with those defaults. A window of M minutes and C channels gives 2 · M · C local tokens, 16 · C temporal tokens and 64 global ones. For 30 minutes and 3 channels: 180 + 48 + 64 = 292. The patch count p appears nowhere. Step through the stages and watch the grid collapse.

Change the channels and the total moves; the patch count never enters it. With the stretch switched on, global query 0 puts most of its attention on ECG II minutes 20 to 29. With it off, the wash goes flat. One learned question found the one thing in the window it was trained to look for.

Here is the counting rule, derived in six steps.

1. One query. Its answer is a weighted sum over all n inputs: one vector, whatever n is.
2. Stack the queries. N queries give an N × D answer. The input count appears only inside the sums.
3. Local stage. For each channel-minute, 2 queries read that cell's patches: 2 per cell, M · C cells, 2MC tokens.
4. Temporal stage. For each channel, 16 queries read its 2M local tokens: 16C tokens.
5. Global stage. 64 queries read all M · C · p patches: 64 tokens.
6. Join them. 2MC + 16C + 64. Doubling p changes none of these terms.

What it buys, measured

The paper timed both models on one GH200 GPU, one window at a time, counting the encoders, the backbone and the heads. In emergency and intensive care, the sensor tokens fell from 5,801.0 to 400.4 per window: 14.49 times fewer. The model ran 3.4 times faster.

In the operating room they fell from 3,915.8 to 838.0, 4.67 times fewer, for a 1.7 times speedup. For the pump they fell from 33.0 to 21.0, 1.57 times fewer, and the model was no faster at all.

Why is 14.49 times fewer tokens only 3.4 times faster? Because the sensor tokens are not the only cost. The encoders still read every patch. The 24 layers' feed-forward parts cost time for every token, sensor or text. The prompt's own tokens do not shrink. Compression removes one cost and leaves the others, so the speedup is smaller than the compression.

Does OpenSLA-H also decide better? Sometimes, and the honest answer is mixed. On surgery set A its act-or-wait score is 80.6 against the base model's 74.2; on surgery set B, 66.3 against 59.0. On the emergency set the two are level: 72.2 against 72.5. On pump set B's what-kind call, OpenSLA-H is worse: 53.9 against 56.3.

Its captions read numbers better, though. When it writes the breathing rate it measured from the wave, its average miss is 1.17 breaths a minute, against 1.59 for the base model. So OpenSLA-H buys speed wherever inputs are long, accuracy in the operating room, and sharper captions.

worked exampleMemory size with the code defaults: 2·M·C + 16·C + 64
  30 min, 3 channels       180 + 48 + 64        = 292 tokens
  30 min, 4 channels       240 + 64 + 64        = 368 tokens
  against a flat prefix    5,400 ÷ 292          = 18.49 times fewer   (toy, Chapter 3)
Measured (Table 21, OpenSLA-B to OpenSLA-H)
  emergency, intensive     5,801.0 ÷ 400.4      = 14.49 times fewer, 3.4 times faster
  operating room           3,915.8 ÷ 838.0      =  4.67 times fewer, 1.7 times faster
  insulin pump                33.0 ÷ 21.0       =  1.57 times fewer, no faster

Now build it yourself. The lab below wires the three stages in numpy; your job is the one line that makes a query look where it should.

A memory that made the pump no faster

What the team sees. A team adds the memory to every model they run, the pump model included. On the pump, nothing gets faster.

What they measure. They count tokens before and after, and time each stage. The pump's 33 tokens became 21, and the time stayed the same: the paper's own table shows 1 times. Running three stages of cross-attention costs about what it saves when the input is already short.

What they change. They skip the memory for short inputs and keep it for long ones. That rule is our reading of the table, not a claim the paper makes.

A nurse's decision comes in steps. First: do we do anything right now? Then: what kind of thing, a medicine, a test, fluids? Then: which one exactly? A wrong "no" at the first step ends the conversation; nobody asks the other two questions.

OpenSLA answers the three questions with three small readers attached to the same final summary. One is a yes-or-no switch. One is a set of switches, one for each kind of action. One is a matcher that compares the summary with each action's written name.

One summary vector

Recall Chapter 3's row: sensor tokens (or the memory's tokens), then the prompt with the person's context and the question. The language model reads the whole row. The hidden state at the very last position of the prompt has, by then, looked at everything before it. Call that vector h. It holds 2,048 numbers: the model's summary of the signals and the story together.

A head is a small layer on top of the network that turns h into an answer, like a dial that reads one gauge. OpenSLA puts three heads on the same h. That one fact explains a lot of what follows: the three answers are read from one shared summary, so whatever h knows, all three can use.

Rung 1: the gate

The necessity head is one straight-line layer. It multiplies h by a learned weight vector, adds a learned bias, and gets a single number. That raw number is a logit: the score before squashing, like a judge's raw points before they become a grade.

The sigmoid squashes any number into the range 0 to 1, like a dimmer with soft ends. The result is the probability that someone acts. At inference, the gate opens when that probability reaches the threshold, the pass mark: 0.5 in the released code. A logit of 1.2 gives 1 / (1 + e^−1.2) = 0.769, so the gate opens.

p_nec

The probability that someone acts in the target interval.

σ

The sigmoid, 1 / (1 + e^−z): any number in, a number between 0 and 1 out.

w, b

The head's learned weights (2,048 of them) and its bias. w · h + b is the logit.

h

The hidden state at the last input position: 2,048 numbers.

Rung 2: one switch per kind

The category heads work the same way, with one independent switch per kind of action. Each kind gets its own logit and its own sigmoid. The switches do not compete, so several kinds can be on at once. That is called multi-label: tick every box that applies. A window can need fluids and an antibiotic in the same five minutes.

The released code turns those probabilities into a choice with three rules. Keep every kind at 0.5 or above. Sort them, most likely first, and keep at most 5. And if the gate is open but no kind reaches 0.5, keep the single most likely kind anyway: the model has already said "act", so it must say what kind.

Rung 3: a name matched by meaning

The third head is different, and it is the clever part. It does not have one output per drug. Instead, it reads each candidate action's name as text. That written name is the action's descriptor, like the label on a jar.

For each name, the head looks up the input vectors of its tokens in the language model, averages them (using at most 64 tokens), and passes the average through a small learned projection. That gives one vector per name, e_j. Meanwhile h goes through its own small network to become a query vector, q(h).

Then it compares them by cosine similarity: how closely two vectors point the same way, 1 when they point the same way and 0 when they are unrelated. Think of two compass needles agreeing. The cosines are divided by a temperature of 0.07 and passed through softmax. Temperature is a contrast knob: it sharpens or flattens the probabilities, but never changes their order. The released code keeps the top 3 names in each chosen kind.

s_j

The cosine between the window's query and name j: between −1 and 1.

q(h)

h after a small learned network: the window's question, in the names' space.

e_j

Name j's vector: the mean of its token vectors, then a learned projection.

P(j)

The probability of name j among the names of its kind.

τ

The temperature, 0.07 in training. Smaller makes the winner stand out more.

Climb the whole ladder in the device. Move the gate, raise the kind bar, change the temperature, and read the record the code would return.

Three things to notice. Below 0.5 the whole ladder under the gate goes grey: nothing is asked. Raise the kind bar past 0.65 and no kind passes, so the code keeps antibiotics alone. And the temperature moves the probabilities a lot while the order of the names never changes.

Why names instead of one output per drug

Chapter 0's closed-list argument comes due here. A classifier needs one output, with its own weights, for every drug on the list. The name head needs only text. A new medicine is a new descriptor: average its token vectors, project, compare. No new weights. The list of answers becomes an open label space, a menu where you can write in a dish. Chapter 11 tests this on an antibiotic the model never trained on.

The pumps are the exception. Insulin has no names to match, only the four dose bins of Chapter 1, so the pump model has rungs 1 and 2 and no rung 3.

How the heads learn

Training needs a loss: one number that says how wrong the model was. Training nudges the parameters to lower it, like improving a golf score. Each rung has its own loss.

The gate uses binary cross-entropy: minus the log of the probability the model gave to the true answer. It is the surprise you feel at the truth. The kind switches use it too, with positive-class weighting: each rare "yes" example counts more, so rare kinds are not ignored. The name head uses cross-entropy over the softmax of the cosines, among the names of the true kind. The three losses add up:

L_act

The whole ladder's loss (the paper's Equation 2).

L_nec

Binary cross-entropy of the gate: −[y log p + (1 − y) log(1 − p)].

L_cat

The same per kind, with each "yes" term multiplied by a weight w_pos.

L_label

−log P(true name), within the true kind.

λ

How much each rung counts. The paper does not publish their values.

The whole ladder, derived in five steps:

1. Gate. z = w · h + b, one number; p = 1 / (1 + e^−z).
2. Kinds. z_k = w_k · h + b_k for each kind k; p_k = σ(z_k), each independent of the others.
3. Names in kind k. e_j = projection of the mean token vector of name j; s_j = cos(q(h), e_j); P(j) = softmax of s_j / τ.
4. Losses. Cross-entropy on each rung, rare kinds up-weighted.
5. Total. L_act = λ_nec L_nec + λ_cat L_cat + λ_label L_label.

worked exampleRung 1, a gate logit of 1.2
  p        = 1 / (1 + e^−1.2) = 1 / 1.301          = 0.769   → open (at or above 0.5)
  loss     if the truth is yes: −ln 0.769           = 0.263
           if the truth is no:  −ln (1 − 0.7685)    = 1.463

Rung 2, toy logits 0.6, −1.1, −2.0, 0.3, −2.0, −0.4
  antibiotics 0.646 · antiemetic 0.250 · analgesic 0.119
  fluids 0.574 · imaging 0.119 · lab test 0.401
  at the 0.5 bar: antibiotics (0.646), then fluids (0.574)

Rung 3, antibiotics, toy cosines 0.62, 0.55, 0.31
  τ = 0.07   divide: 8.857, 7.857, 4.429    softmax: 0.725, 0.267, 0.009
  τ = 1      softmax of 0.62, 0.55, 0.31:            0.375, 0.350, 0.275
  loss if the first is right: −ln 0.725 = 0.322 at τ = 0.07, −ln 0.375 = 0.981 at τ = 1

Positive weighting (illustrative)
  a kind used in 1 window of 50 → w_pos = 49: its rare yes-windows weigh as much as all its no-windows

A gate of 0.46 is a logit of ln(0.46 ÷ 0.54) = −0.16

Look at what the temperature does to learning. At τ = 1 the three cosines give an almost three-way tie, so the right answer earns only 0.375 and the loss stays high whatever the cosines do. At τ = 0.07 a gap of 0.07 in cosine becomes a gap of 1 before the softmax. The model is rewarded for small, real differences in meaning.

What the code returns

The device's record uses the released code's field names exactly. necessity_probability and action_needed are rung 1. category_probabilities holds every kind's probability, and selected_categories the kinds the rules kept, most likely first. fine_labels lists, for each kept kind, its top names, each with its cosine_similarity. caption holds the text the decoder writes, which Chapter 6 builds.

Now write the rules yourself. The lab hands you the heads' numbers; you write the gate rule, the fallback and the ranking, then send 400 toy windows down the ladder.

A model that never suggests anything for calm patients

What the ward reports. "It never suggests anything for patients who look calm and then get a test."

What you measure. On those windows the gate's probability sits between 0.30 and 0.45. The kind scores are never even read, because the code chooses nothing below a shut gate. Errors flow down the ladder. In Lab 2's toy, the gate is right on 0.90 of the windows that needed an action; the right kind is on top in 0.81; the right name too in 0.73. Each rung can only lose windows.

What you change. You have two honest choices. Lower the gate threshold, and accept more alarms (Chapter 12 counts them). Or show the kind scores for review even when the gate is shut, so a person sees what the model would have said. The ward and its numbers are illustrative.

One last detail matters when you read the paper's tables. The name rung is scored only inside kinds that were truly used. So a published name score leaves out every gate and kind mistake. That is fair for comparing models, and optimistic for a real ladder, where the rungs' errors multiply.

Ask a colleague why she gave a medicine. She does not narrate the whole shift. She picks the findings that bear on what she did: "pain score of 5, nausea since arrival, breathing fast." A note written with the decision in mind reads very differently from one written blind.

OpenSLA writes its caption the same way. In plain words: the caption writer gets two extra inputs, what the model has just decided, and the sensor tokens most relevant to that decision. Two dials, which start almost closed, set how much each input changes the writing.

Writing one token at a time

A language model writes autoregressively: one token at a time, each based on the ones before. It is like dictating a letter word by word. To write "Respiratory rate was elevated", it writes "Respiratory", then reads that and writes "rate", and so on.

Call the hidden states at the caption's positions H, one row per caption token. Normally each row goes straight to the model's output layer, the LM head, which turns it into a probability for every possible next token. OpenSLA adds one step in between. The part that writes the caption, with its extra inputs, is the fusion decoder: the writer at the end of the line.

The two extra inputs

The first input is the action profile, P: a few tokens that carry what the model just decided and how sure it was. Think of a sticky note with the verdict on it. There is one token for the gate and one for each kind of action. Each token is that action's embedding plus its predicted score, projected into the model's width.

The released code keeps the profile short: at most 3 kinds, and at most 3 facts per kind. It uses its own thresholds for what goes in: 0.5 for the gate, 0.15 for a kind, 0.5 for a fact. Notice the 0.15. A kind does not need to pass the decision bar to be mentioned in the reasons.

The second input is the sensor evidence, S: the sensor tokens most relevant to the decision, fetched for the writer. Think of the exhibits pulled for a hearing. The model retrieves the sensor tokens that score as most relevant given H and P, projects them into the model's width, and weights them by their retrieval scores. OpenSLA-H can fetch up to 16 detail tokens per wave channel and 2 per charted measurement.

Equation 1: two gated corrections

Here is the decoder in one line, the paper's Equation 1.

H

The caption's hidden states from the language model, one row per caption token.

f_p, f_s

Two cross-attention branches (cross-attention, then a feed-forward block, then an output projection). Queries come from H; f_p reads P, f_s reads S.

P

The action profile: a gate token and one token per kind, each an embedding plus its predicted score.

S

The retrieved sensor evidence, weighted by retrieval score.

g_p, g_s

Learned gates between 0 and 1: how much of each correction gets in.

H′

The corrected states. They go to the original LM head, which writes the caption.

Read it as an edit. The first term, H, is what the language model would have written anyway. Each branch computes a residual: a correction added to what is already there, like an editor's tracked change. Each correction passes through a gate: a learned dial between 0 and 1 that sets how much of it gets in, like a volume fader.

Why the dials start almost closed

In the released code, each gate is the sigmoid of a learned number that starts at −3. And each branch's output projection starts as all zeros. Put those together and look at the very first training step.

g

Either gate, g_p or g_s.

a

The gate's learned number. Training can push it up, opening the gate, wherever that lowers the loss.

With zero output projections, f_p and f_s are exactly zero at step 0. So H′ = H + 0.047 × 0 + 0.047 × 0 = H. The pretrained writer is untouched. Training starts from a language model that already writes good English, and grows the new paths only as far as they help. Try both states in the device.

At the start, the gates can move and nothing changes: zero times anything is zero. Once trained, opening the action gate pushes the next word toward the decision's vocabulary, "elevated". Opening the evidence gate pushes it toward the measured number, "26".

The decoder in five steps:

1. Start with H. The caption states, one row per caption token.
2. Branch p. Queries from H read P, the handful of action tokens: f_p(H, P), the same shape as H.
3. Branch s. Queries from H read S, the retrieved evidence: f_s(H, S), the same shape as H.
4. Gate and add. H′ = H + g_p f_p + g_s f_s.
5. At step 0. g = σ(−3) = 0.047 and the output projections are zero, so f_p = f_s = 0 and H′ = H.

One loss for everything

The whole model trains on one objective: the ladder's loss from Chapter 5, plus a loss for the caption.

L_act

The decision ladder's loss from Chapter 5.

L_text

Cross-entropy on each caption token. Tokens about the signals get a higher weight.

λ_text

How much the caption counts. The paper does not publish its value.

L

The single number training lowers.

Two training details matter. First, teacher forcing: during training the writer sees the true caption so far, not its own guesses, like tracing over a model answer. Second, the action profile it reads holds the model's predicted actions, not the true ones. So the writer learns to explain whatever the model decided. Keep that in mind; it comes back in the story below.

Changing a big model cheaply: LoRA

Fine-tuning means changing a pretrained model's weights for a new job, like retraining a chef for a new cuisine. Changing all 2 billion parameters is costly and can wreck what the model already knows. LoRA (low-rank adaptation) is the cheap way. It keeps each big weight matrix W frozen and learns a thin add-on beside it. Think of clip-on lenses instead of new eyes.

W

A frozen weight matrix of the language model, for example 2,048 × 2,048.

B A

The learned add-on: A is r × 2,048 and B is 2,048 × r, so their product has W's shape.

r

The rank: how thin the add-on is. 16 here.

α

A scale for the add-on's effect: 32 here, so α / r = 2.0.

W′

The weight the model actually uses.

Count what that saves for one 2,048 × 2,048 matrix. Full fine-tuning would change 4,194,304 numbers. LoRA at rank 16 learns 16 × (2,048 + 2,048) = 65,536 of them: 1.56%. The paper's settings are rank 16, α 32 and a dropout of 0.05.

So what trains and what stays frozen? The LoRA add-ons on the language model train, and so do the sensor memory and the decoder. The wave encoder and the rest of the language model stay frozen.

The recipe is the same for every trained model in the paper's contest, rivals included. AdamW optimiser; a peak learning rate of 5 × 10⁻⁵; weight decay 0.01; a warm-up over the first 10% of training, then a linear decay; gradients clipped at 0.5; 16 windows per step in every setting. Training is capped at 24 hours on H200 GPUs.

worked exampleGate at the start of training
  σ(−3) = 1 / (1 + e^3) = 1 / 21.09                   = 0.0474
  H′    = H + 0.0474 × 0 + 0.0474 × 0                 = H

LoRA on one 2,048 × 2,048 matrix (rank 16, α 32)
  frozen        2,048 × 2,048                         = 4,194,304 numbers
  learned       16 × (2,048 + 2,048)                  =    65,536 numbers
  share         65,536 ÷ 4,194,304                    = 1.56%
  scale         α ÷ r = 32 ÷ 16                       = 2.0

One pass over the emergency and intensive-care training windows
  106,596 windows ÷ 16 per step                       = 6,662.25, about 6,662 steps

A caption that explained a wrong decision

What the team sees. On one window the gate wrongly said "yes, antibiotics". The caption, calm and fluent, lists evidence for antibiotics.

What they measure. They compare each caption's action facet with the recorded action and with the predicted action. It agrees with the prediction, not the record. That is exactly what training asked for: the writer reads the predicted profile, so it explains the prediction, right or wrong.

What they change. They always show the gate's probability next to the caption. And they read the caption as the model's reasons, not as proof. The paper itself calls these explanations evidence associations, not causes. The case is illustrative; the mechanism is the paper's.

Does the decoder help the decision itself, or only the words? Chapter 10 measures it. On surgery set A's act-or-wait call, a model with no decoder scores 78.3, one whose decoder ignores the predicted actions scores 80.4, and the full model 83.4, on a scale where a coin scores 50.

A ward has 100 windows, and 10 of them are followed by an action. A model that always says "no action" is right 90 times out of 100. Is it a 90% model?

Of course not: it never catches a single action. It is a model that has learned the ward is usually quiet, and nothing else. This chapter builds a score that cannot be fooled that way. In plain words: balanced accuracy averages how many real actions a model caught with how many quiet windows it left alone. So a model that always says the same thing scores 50, like a coin.

Four ways to be right or wrong

Every yes-or-no answer lands in one of four boxes. Count them for a whole test set and you have a confusion matrix: a 2 × 2 scorecard. In ward words:

• Caught (a true positive, TP): the window needed action, and the model said act.
• Missed (a false negative, FN): the window needed action, and the model said wait.
• Left alone (a true negative, TN): a quiet window, and the model said wait.
• False alarm (a false positive, FP): a quiet window, and the model said act.

Think of a smoke alarm. It can notice a fire, sleep through a fire, stay quiet on a quiet night, or go off over toast.

Plain accuracy is the share of all answers that were right: (TP + TN) divided by everything. It has the flaw from the opening. When quiet windows are common, saying "quiet" every time scores high.

Two rates and their average

Split the question in two. Sensitivity asks: of the windows that needed action, what share did the model catch? That is how many fires the alarm noticed. Specificity asks: of the quiet windows, what share did it leave alone? That is how many quiet nights it stayed silent. Balanced accuracy is their average.

TP

Caught: windows that needed action, answered "act".

FN

Missed: windows that needed action, answered "wait".

TN

Left alone: quiet windows, answered "wait".

FP

False alarms: quiet windows, answered "act".

Sens

Sensitivity: the share of real actions caught.

Spec

Specificity: the share of quiet windows left alone.

BAcc

Balanced accuracy, reported out of 100 in the paper's tables.

Now the lazy model. Always "no" on 10 action windows and 90 quiet ones: it catches 0 of 10, so its sensitivity is 0. It leaves all 90 quiet windows alone, so its specificity is 1. The average is (0 + 1) / 2 = 0.5. Its plain accuracy was 0.90; its balanced accuracy is 0.50.

Why exactly 50? Any answer that never changes gets 1 on one rate and 0 on the other, so it averages to 0.5. "Always yes" catches every action and leaves no quiet window alone: (1 + 0) / 2, the same 0.5. A fair coin gets about 0.5 on both rates. So on this scale, 50 is chance: what guessing earns. Think of a test marked so that guessing earns half.

Try to fool it. Pick a lazy model and slide the share of windows that need action.

Accuracy rewards "always no" whenever actions are rare: at 10%, it scores 90. Balanced accuracy pins every constant answer at 50, and the coin near it. Slide to 50% and watch the two scores meet.

Why the paper's test sets are half and half

The paper's test windows are balanced for necessity: as many with an action as without. With equal counts, accuracy and balanced accuracy are the same number. Derive it. Call the number of action windows N, and the number of quiet ones N too. Then accuracy = (TP + TN) / 2N = (TP/N + TN/N) / 2. And TP/N is the sensitivity, TN/N the specificity. So accuracy equals their average.

So for the act-or-wait call, the score is also simply the share of right answers. The kinds and the names are where balance matters. They are rare: in the emergency set, a kind of action has as few as 12 positive test windows, and the middle kind has 41. Against thousands of windows where that kind was not used, balanced accuracy is the honest score.

Where the threshold comes from

Every yes-or-no needs a threshold, like the gate's 0.5. In the paper's result tables, each threshold is picked to maximise balanced accuracy on the validation set: held-out windows used to pick settings, like a practice exam. It is done separately for every yes-or-no target. The emergency set has 403 validation windows for this.

A score that needs no threshold

A second score skips the threshold entirely. AUROC (area under the ROC curve) is the chance that a randomly picked action window gets a higher score than a randomly picked quiet one. Think of a ranking contest of pairs. A model that ranks every action window above every quiet one scores 100. Random scores give 50.

AUROC

Reported out of 100. Ties count as half a win.

action window

One drawn at random from the windows that needed action.

quiet window

One drawn at random from the windows that did not.

AUROC needs scores, not just answers. The chat models in Chapter 8 return answers without probabilities, so they get balanced accuracy but no AUROC; the tables show a dash.

Averaging, and the ±

With many kinds, the paper computes each kind's score and then averages them. That is a macro-average: every kind counts equally, however rare. Think of one vote per class, not one per student. Names are scored inside the kinds that were truly used, averaged within each kind, then across kinds.

Every score in the tables carries a ±. It comes from the bootstrap: re-draw the test people with replacement 2,000 times, rescore each time, and see how far the score moves. Think of reshuffling the deck to see how lucky a hand was. The paper resamples people, not windows, and for surgery set B it resamples surgical cases.

The ± is half the width of the resulting 95% confidence interval. It measures how much the score depends on which patients happened to be tested. It does not measure training luck: each model was trained once.

The whole chapter in five steps:

1. Two rates. Sens = TP / (TP + FN); Spec = TN / (TN + FP).
2. Average them. BAcc = (Sens + Spec) / 2.
3. Constant answers. "Yes" gives Sens 1, Spec 0, so 0.5. "No" gives 0 and 1, so 0.5.
4. Balanced sets. With as many action windows as quiet ones, Acc = (TP/N + TN/N) / 2 = BAcc.
5. AUROC. Over all action-quiet pairs, the share where the action window scores higher; ties count half.

worked example1,000 test windows, 500 with an action and 500 without (counts illustrative)
  caught 360 · missed 140 · left alone 365 · false alarms 135
  sensitivity        360 ÷ 500                    = 0.720
  specificity        365 ÷ 500                    = 0.730
  balanced accuracy  (0.720 + 0.730) ÷ 2          = 0.725
  accuracy           (360 + 365) ÷ 1,000          = 0.725   (the same: the set is balanced)

Always "no" on 100 windows, 10 with an action
  accuracy           90 ÷ 100                     = 0.90
  balanced accuracy  (0 + 1) ÷ 2                  = 0.50

AUROC by hand (scores illustrative)
  action windows score 0.9, 0.7, 0.4 · quiet windows score 0.6, 0.3, 0.2
  0.9 beats 3 of 3 · 0.7 beats 3 of 3 · 0.4 beats 2 of 3
  wins               8 of 9 pairs                 = 0.889, an AUROC of 88.9

Reading a ± (act or wait, balanced accuracy)
  surgery A   OpenSLA-H 80.6 ± 2.8  →  77.8 to 83.4
              best rival 71.5 ± 3.0 →  68.5 to 74.5   no overlap
  pump A      OpenSLA-B 65.0 ± 1.3  →  63.7 to 66.3
              best rival 62.9 ± 1.5 →  61.4 to 64.4   they overlap

Overlapping intervals are a rough check, not a formal test. Read them as "close", and read separate intervals as "unlikely to be the luck of who was tested".

Now build the score yourself. The lab sweeps a threshold across a toy ward and plots both scores; your job is the balanced one.

Rivals that scored a coin while looking clever

What you see. On surgery set B, every rival's act-or-wait balanced accuracy sits between 48.9 and 52.1. Yet some of the same rivals score well on the kind of action: the model trained from scratch reaches a kind AUROC of 70.9.

What you measure. A balanced accuracy this close to 50 means the yes-or-no answer carries almost no information. The model is guessing when to act, even if it knows what kind of thing happens in an operation.

How to read it. Kinds may be learnable from the stage of the surgery, while the exact timing is not. That is our reading; the paper says only that acting from the sensors alone is hardest in the operating room. The habit to keep: treat any score within its ± of 50 as a coin.

The paper runs a contest. Every model gets the same test windows. Every model that trains gets the same training data and the same compute budget, and training is capped at 24 hours on H200 GPUs. Where a rival reads signals with a pretrained encoder, it gets the same pretrained encoders as OpenSLA. Then all of them answer the same three questions on six groups of patients.

Here is the result in plain words. OpenSLA made the best act-or-wait call in all six groups and the best what-kind call in five. A chat model named the exact action better in two groups. And a small model trained from scratch picked the insulin dose range better in one.

The rivals

A baseline, or rival, is another model scored on the same test, to compare against: the other runners in the race. The paper uses nine, in four families.

• Trained from scratch. Two models whose weights start random and learn only from this data, like a new hire with no experience.
• Signal foundation models. A foundation model is pretrained on huge amounts of data and reused for many jobs, a generalist with a broad education. Three of them, built for time series and glucose, are kept frozen. Only a linear probe is trained on top: a single straight-line layer that reads the frozen model's features, like asking one quick question of an expert who will not study more.
• Signal readers. Two sensor-language models, built to describe signals in words, adapted on this paper's training data.
• Chat models that run code. Two very large multimodal chat models, used zero-shot: with no training on this task at all, like sitting an exam cold.

The chat models need a word on how they see the signals. Thousands of numbers pasted into a prompt would be unreadable. Instead they work in a CodeAct style: they write and run small Python programs that inspect the measurements and compute summaries. Think of a consultant who asks for the spreadsheet. They return answers, not probabilities, so they get balanced accuracy but no AUROC.

How to read the tables

Chapter 7 gave you everything you need. Each row of the paper's tables is a model. The columns are the three rungs: act or wait, what kind, which one. Each cell is a score out of 100 with its ±, and a dash means that model gives no score of that type. The device below holds every balanced-accuracy cell that matters, one group and one rung at a time.

Act or wait: best in all six

Take the first rung group by group, best OpenSLA model against the best rival. Emergency: 72.5 against 64.6, a margin of 7.9 points. Intensive care: 75.4 against 71.2, 4.2 points. Surgery A: 80.6 against 71.5, 9.1 points. Surgery B: 66.3 against 52.1, 14.2 points. Pump A: 65.0 against 62.9, 2.1 points. Pump B: 75.5 against 72.8, 2.7 points.

Not every margin is equally solid. Chapter 7 gave you the tool: turn each score into its interval. Surgery A's 80.6 ± 2.8 runs from 77.8 to 83.4; the rival's 71.5 ± 3.0 runs from 68.5 to 74.5. They do not touch, so the margin is unlikely to be the luck of who was tested. Pump A's 65.0 ± 1.3 and 62.9 ± 1.5 overlap: call that one close.

What kind: best in five

On the second rung, OpenSLA leads in five groups. It loses on pump set B, where a small model trained from scratch picks the dose range at 66.2 against OpenSLA-B's 56.3: 9.9 points behind.

Which one: two cells go to a chat model

On the third rung, a chat model wins twice. In intensive care it names the action at 79.5 against OpenSLA's 76.5, 3.0 points ahead. In surgery set A it scores 69.2 against 67.6, 1.6 points ahead. One reading, ours and not the paper's: once the kind is known, naming the exact drug leans on medical knowledge and on the notes, and a very large chat model is strong at both.

The pattern across settings

The paper draws three lessons from the whole table. First, acting from the sensors alone is hardest in the operating room. On surgery set A, four rivals score between 45.0 and 48.0 on act or wait, at or below a coin.

Second, OpenSLA's gains are largest in the hospital and the operating room, where the notes carry the most. Third, the insulin dose range stays hard for everyone. A dose depends on the carbohydrates the person is about to eat, and the window cannot contain a meal that has not happened yet. On pump set A, both chat models sit at a coin: between 49.9 and 51.2.

How to tell a margin from luck

Here is the check in three steps, applied to any cell.

1. Make intervals. Each score becomes score − ± to score + ±.
2. Compare them. If OpenSLA's interval sits entirely above the rival's, the margin is unlikely to be luck of the test sample.
3. Otherwise, call it close. Overlap does not mean equal; it means this test cannot tell them apart.

worked exampleAct-or-wait margins, best OpenSLA minus best rival (balanced accuracy)
  emergency        72.5 − 64.6   =  7.9
  intensive care   75.4 − 71.2   =  4.2
  surgery A        80.6 − 71.5   =  9.1
  surgery B        66.3 − 52.1   = 14.2
  pump A           65.0 − 62.9   =  2.1
  pump B           75.5 − 72.8   =  2.7
Where a rival wins
  which one, intensive care      79.5 − 76.5   = 3.0
  which one, surgery A           69.2 − 67.6   = 1.6
  what kind, pump B              66.2 − 56.3   = 9.9
Intervals
  surgery A   80.6 ± 2.8 → 77.8 to 83.4    rival 71.5 ± 3.0 → 68.5 to 74.5    separate
  pump A      65.0 ± 1.3 → 63.7 to 66.3    rival 62.9 ± 1.5 → 61.4 to 64.4    overlap

A score below a coin

What you see. The model trained from scratch scores 48.0 on surgery set A's act or wait. That is below 50.

What it means. A score below 50 is no better than chance. Either the threshold picked on the validation windows did not carry over to the test windows, or the input held no usable signal. Surgery set A has waves in only 25.1% of its training windows, so most windows give a sensor-only model little to work with.

The habit. Read any score within its ± of 50 as a coin, whoever produced it.

Two changes would sharpen this contest. Probabilities from the chat models would give them an AUROC too. And several training runs per model would show how much each score moves between runs; today's ± covers only which patients were tested.

Here is a real window from the paper. The monitor's heart-rate readings over the window: median 100, ranging from 95 to 108, 30 readings in all. A chat model is asked what is happening and writes: "HR approximately 90 bpm." OpenSLA writes: "HR median 100, range 97-104, n=30. ECG Lead II: irregular Lead II rhythm proxy; rate proxy 95.2/min, IQR 0.40."

One of those captions is checkable and right. The other sounds fine and is wrong by about ten beats a minute. This chapter is about telling them apart. In plain words: if a caption is good, you can read its numbers back out, and they match the monitor.

Pulling numbers out of words

The paper scores captions in two ways. The first is state understanding: whether the numbers in a caption match what the window really shows. Think of checking a weather report against the thermometer.

A program, a parser, reads each generated caption and pulls out one specific quantity. It compares that number with the reference statistic: the true number computed from the window, the answer key. Because the training captions were built by rules from extracted facts (Chapter 2), the model learned to write numbers in fixed, parseable forms. That is what makes this check possible at all.

Eight quantities are scored, chosen by setting:

• Emergency and intensive care: the cardiac index (how much blood the heart pumps per minute, scaled to body size; like engine output per kilo of car), the ECG lead II rate, and the breathing rate measured from the wave.
• Operating room: the breathing rate, the temperature, and the end-tidal CO₂: the carbon dioxide in each breath out, a sign of how well breathing works under anaesthesia. Think of the exhaust reading of the lungs. All three are window medians.
• Insulin pump: the mean basal rate, in units an hour, and the most recent carbohydrate amount in the last 120 minutes, in grams.

The average miss

For each test window, take the size of the miss: the parsed number minus the true one, ignoring the sign. Then average those sizes over all the windows scored. That is the mean absolute error (MAE). Think of how far, on average, your darts land from the bullseye, whichever side they land on.

MAE

The mean absolute error, in the quantity's own unit. Smaller is better; 0 is perfect.

ŷ_i

The number parsed from the caption for window i.

y_i

The reference statistic computed from window i.

N

The number of windows scored.

By hand, with three toy windows: true breathing medians 26, 18 and 22; parsed 27, 18 and 20. The misses are 1, 0 and 2. Their average is 3 ÷ 3 = 1.0 breath a minute.

Guess before you look. How far off do you think OpenSLA's captions are, on average, for each quantity?

Breathing from the wave is the headline. OpenSLA-H misses by 1.17 breaths a minute on average; the best rival, a signal reader, by 5.87. That is about five times smaller. The other results tell the same story with different sizes. End-tidal CO₂: 3.18 against 5.47. Carbohydrates: 0.38 grams against 1.20. ECG heart rate: 15.9 against 20.5.

Now the honest cells. OpenSLA-B, not OpenSLA-H, wins the cardiac index: 0.93 against the best rival's 1.13. On the basal rate, a chat model ties OpenSLA-B at 0.07. And the ECG-rate miss of 15.9 beats a minute is still large, even though it is the best in the table.

A word on how to say these comparisons. The project page says "next best" when it compares OpenSLA-H with a rival. But OpenSLA-B often sits between the two: on CO₂ it scores 5.13, between 3.18 and 5.47. So this lesson says "best rival", meaning the best model that is not OpenSLA.

worked exampleMAE by hand (illustrative)
  true      26   18   22
  parsed    27   18   20
  misses     1    0    2      MAE = 3 ÷ 3 = 1.0 breath a minute

How many times smaller than the best rival (Table 5)
  breathing from the wave    5.87 ÷ 1.17   = 5.02
  end-tidal CO2              5.47 ÷ 3.18   = 1.72
  carbohydrate amount        1.20 ÷ 0.38   = 3.16
  ECG heart rate             20.5 ÷ 15.9   = 1.29

The second score: does the wording match?

The second way of scoring is about the explanation, the action facet. Here there is no single number to parse, so the paper compares wording. BERTScore compares two texts by how close the meanings of their words are, using another language model's vectors. Think of grading an essay by its paraphrase, not its spelling.

It works like this. Each word of the generated text is matched to its most similar word in the reference text, by cosine, as in Chapter 5. The average of those best matches is the precision: how much of the candidate matches the reference. Doing it the other way round gives the recall: how much of the reference is covered. Their balance is F1.

P

Precision: the average best match of each candidate word.

R

Recall: the average best match of each reference word.

F1

Their balance. The paper reports the mean per-window F1 × 100, using DistilBERT's vectors, against the action-evidence part of the reference caption.

The results are close. On the emergency set, OpenSLA-H scores 82.0, OpenSLA-B 81.5, and the best signal reader 79.5. In intensive care, 80.9 against 80.3. On surgery set A a signal reader edges ahead: 81.7 against OpenSLA-B's 81.5 and OpenSLA-H's 81.0. On surgery set B, OpenSLA-B scores 80.9 and the other two tie at 80.7. On the two pump sets, OpenSLA-B scores 85.6 and 87.6, against 84.4 and 83.2. The paper's own phrase is "strongest overall agreement", and that is fair: not every cell.

Does the caption follow the signal, or only the story?

A caption could sound right by echoing the notes. The paper checks this with sensor substitution: keep the story, swap the recording, and see what changes. Think of changing one ingredient and tasting again.

In its example, the patient's context stays the same while a recording with higher arterial pressure is swapped in. The finger-clip rate in the caption moves from 76.1 to 82.1 beats a minute, and a "high-rate ECG" finding appears. The overall concern stays the same. The numbers follow the signal; the story still frames them.

A caption that sounds right

What the team sees. A team's caption model scores well on wording, but nurses keep spotting wrong numbers.

What they measure. They extract the numbers and compute the MAE separately. Then they look at why the wording score missed it. Take the reference "HR median 100" and the candidate "HR median 90". By plain word overlap, 2 of 3 words match each way, so P = 2/3, R = 2/3 and F1 = 0.667. A meaning-based score does even less to separate them, because "90" and "100" mean nearly the same kind of thing.

What they change. They always pair a wording score with a number check. The example and its numbers are a toy; the blind spot is real.

One limit applies to the number check too. It can only score captions that state the number. A model that never writes a heart rate never gets an ECG-rate score. Where Table 5 shows a dash, no score was reported for that model.

A cake recipe works. Which ingredient matters? The only way to find out is to bake it again without the eggs, then again without the butter, and taste each one.

The paper does exactly that to OpenSLA. It trains versions with one piece removed and checks how well each one decides. In plain words, the answer is: removing the sentences hurts most.

Remove one part, retrain, compare

That method is an ablation: remove one part, retrain, and compare. Every version in this chapter is built on OpenSLA-B and trained the same way. Each is scored by AUROC (Chapter 7) on the two surgery sets: act or wait and what kind, on surgery set A and surgery set B. That gives four numbers per version.

For one column, a part's worth is simply the drop when it goes:

Δ

What the removed part was worth on that column, in AUROC points.

AUROC_full

The full model's score: 83.4 and 80.3 on surgery set A, 74.8 and 85.9 on surgery set B.

AUROC_variant

The score with one part removed and the model retrained.

This works only if exactly one part changed. Then the whole drop belongs to that part.

Four ways to use a language model

Before building OpenSLA, the paper compared four ways to put a language model to work on sensor decisions. First, zero-shot chat models, as in Chapter 8. Second, action-only training: fine-tune the model on the action labels alone, with no sentences. That is the recipe of vision-language-action models: robot models that read camera images and instructions and output robot actions, graded on the actions alone. Think of marking only the final answer.

Third, contrastive alignment: train signals and text to land near each other when they match, like matching socks into pairs. That is the signal readers' recipe. Fourth, joint training on actions and captions, which is OpenSLA. Joint training won. The ablations below show why, piece by piece.

Take one piece out at a time

Here are the versions, with what each one removes.

• Actions only. No captions at all: the model is trained on the ladder's labels alone.
• No conditioning. The decoder is kept, but its action branch is removed, so the writer never sees the predicted actions.
• No decoder. Captions are written straight from the language model, with no fusion decoder.
• No signals. Every sensor input is removed; the text context, heads, captions and decoder stay. This is the context-only variant: judging from the chart notes alone.
• Frozen + projector. The language model is frozen, and a small trainable projector maps the sensor vectors into its input, like an adapter plug. This is the LLaVA-style design.
• Frozen + gated. The language model is frozen, and new attention layers inside it read the sensors, each behind a gate, like side doors with dimmer switches. This is the Flamingo-style design.

Pick each version and watch which column falls.

The pattern is clear. Removing the captions costs the most everywhere: surgery set B's act-or-wait AUROC falls from 74.8 to 54.1, close to a coin. Removing the decoder costs less, and removing only its action inputs less again. And the frozen model with a projector collapses on surgery set A's act or wait: 51.5, a coin.

Why would sentences improve a yes or no?

Here is our reading; the paper reports the effect but does not prove a cause. Think back to Chapter 5: all three heads read one vector, h. Now think about what it takes to write "median 26, range 23 to 30" in the caption. The shared summary has to hold those facts.

A yes-or-no label never forces that. A model trained on labels alone can find a shortcut, a pattern that predicts the label without understanding the window. A model that must also write what it saw cannot skip the window. It is Chapter 2's student again, the one who learns more because she must show her work.

Which kind of sentence matters?

The paper also trained eight mixes of the four caption facets. All four together was best on every column. The action facet mattered most. Adding it to the other three lifts surgery set A's act-or-wait AUROC from 81.6 to 83.4, a gain of 1.8. On surgery set B it lifts it from 64.3 to 74.8, a gain of 10.5. Mix your own.

How the sensors get in

The three ways of feeding sensors to the language model differ a lot. On surgery set A's act or wait, the frozen model with a projector scores 51.5. The frozen model with gated attention scores 82.9. And LoRA on the model itself, OpenSLA's choice from Chapter 6, scores 83.4. On surgery set B's act or wait the gaps are bigger: 54.4, 68.9 and 74.8.

Do the signals matter at all?

This is the most honest number in the chapter. With no sensor input at all, only the text context, the same model reaches 79.8 on surgery set A's act or wait. The full model reaches 83.4. So the signals add 3.6 points there, 4.6 on surgery set A's kind, 9.9 on surgery set B's act or wait, and 2.5 on its kind.

The notes alone are strong. A great deal of what a clinician will do next is written in why the patient is there and what was already done. The signals add real information on top, most of all on surgery set B, but they are not most of the score.

worked exampleWhat each piece is worth, Δ = full − variant (AUROC)
                               A act   A kind   B act   B kind
  full model                    83.4    80.3     74.8    85.9
  captions (actions only)       +5.3    +8.2    +20.7   +13.1
  action conditioning           +3.0    +3.1     +8.5    +1.5
  the decoder                   +5.1    +3.9     +9.7    +1.8
  the other three facets
    (observational only)        +1.9    +4.1    +10.7    +4.8
  the signals (context only)    +3.6    +4.6     +9.9    +2.5
  LoRA over gated attention     +0.5    +4.2     +5.9    +3.8
  LoRA over a projector        +31.9   +13.9    +20.4   +25.9
Action facet added to the other three
  surgery A act                 81.6 → 83.4 = +1.8
  surgery B act                 64.3 → 74.8 = +10.5

A model that writes well and cannot decide

What you see. The frozen model with a projector writes fine captions, yet scores 51.5 on surgery set A's act or wait: a coin.

What you measure. Compare it with the two other designs on the same data: 82.9 with gated attention, 83.4 with LoRA. The data, the heads and the captions are the same. Only the way the sensors get in, and whether the model itself can change, differs.

How to read it. With the language model frozen and the sensors only at its input, the decision never shaped the model. Letting the model adapt fixes it. That reading is ours.

Keep the limits of these studies in view. They use two datasets from one setting, AUROC only, and one training run per version. The ± in the main tables covers which patients were tested, not reruns.

A new antibiotic appears on the ward's list. With a classifier, you would collect labelled windows where it was given, add an output, and retrain. With OpenSLA, you type its name.

This chapter tests whether that really works, and what else the model learned without being asked. In plain words: actions are matched by the meaning of their names, so a new name can be scored the day it is added. And the model's inner summary turns out to hold more than it was trained to say.

A medicine the model never trained on

The paper runs a clean test. It removes one drug, Imipenem/Cilastatin, from the training targets entirely. The antibiotic kind is still seen in training; only this one name is never a target. Call it an unseen action: an action whose label was removed from training. Think of a word you never studied but can guess from its roots.

At test time, nothing is retrained. The descriptor "antibiotic: Imipenem/Cilastatin" is turned into a vector exactly the way Chapter 5's name head does it. Average the language model's token vectors for the text, then apply the trained projection. The score is the cosine between that vector and the window's projected decision vector. The test windows come from the intensive-care set, half where the drug was given and half where it was not.

s_new

The score for the unseen action. Above a threshold, predict "given".

q(h)

The window's decision vector, exactly as in Chapter 5.

proj

The descriptor projection learned in training, applied unchanged.

the new name

The text "antibiotic: Imipenem/Cilastatin". Every parameter stays fixed.

The result: OpenSLA scores an F1 of 70.0 and a balanced accuracy of 71.4. The best chat model scores 42.9 and 61.9; the other chat model 34.5 and 54.8; a signal reader 17.4 and 54.8. That is 9.5 points of balanced accuracy over the best chat model, on a drug OpenSLA never saw as a target.

Where the new name lands

Why does it work? The paper draws the model's decision vectors from the intensive-care set on a flat map, using PCA (principal component analysis): a way to squash long vectors down to two numbers each, keeping as much of their spread as possible, so we can draw them. Think of a shadow that keeps the shape. On that map, the antibiotics form a cluster, and the unseen Imipenem lands near meropenem and the other antibiotics.

Try the same idea on a toy map. Add the new name, then try writing it badly.

A well-written new name lands inside the antibiotic cluster and ranks near the top for this window, with no retraining. The badly written one lands far from every cluster. Its tokens carry little meaning, so its vector points nowhere useful.

A hospital it never saw

The second test is an external cohort: a group of people from a dataset the model never trained on, like an away game. The paper takes 1,000 windows from 87 people in the held-back new-hospital set: 500 with an action, 500 without, on the same five-minute target. No adaptation of any kind.

The act-or-wait call travels well. OpenSLA-B scores 74.40 balanced accuracy (AUROC 81.75) and OpenSLA-H 73.00 (AUROC 80.52). The two signal readers score 69.30 and 67.70.

The what-kind call barely travels. OpenSLA-B scores 56.35 and OpenSLA-H 58.49, against 54.33 and 50.02 for the signal readers. All four are close to a coin. A new hospital records its actions in its own way, and the kinds do not carry over as well as the yes or no.

A space where doses line up

The third test looks inside the model. Take the vectors of two insulin decisions, z_a and z_b, and mix them in steps. A mix is interpolation: α of one vector and 1 − α of the other, like blending two paint colours.

z(α)

The mixed vector at step α.

α

The mix, from 0 (all z_a) to 1 (all z_b).

z_a

One insulin decision's vector: a small dose.

z_b

Another's: a large dose.

For each mix, the paper finds the nearest real window and reports its bolus. At α = 0 it is 0.05 units. At 0.4 it is 1.2 units, at 0.8 it is 7.7, and at 1 it is 14.4. The doses rise in order along the line, although the model was only ever trained on four dose bins.

Reading the next two hours off frozen features

The last test asks whether the pump model's inner summary knows about the future. Here the paper does not generate anything. It trains a separate, small model on top of OpenSLA's frozen features: the model's vectors used as they are, with a small readout trained on top. Think of reading a sealed instrument's dials.

The readout is ridge regression: a straight-line fit with a penalty that keeps the weights small, like a ruler on a spring. From a 2-hour pump window, it predicts four numbers about the next 2 hours of glucose: the mean, the minimum, the maximum and the slope.

X

The frozen features, one row per window.

y

The future glucose summary to predict, such as the next 2 hours' mean.

λ

The penalty. Larger pulls the weights toward zero.

I

The identity matrix: the penalty applies to every weight equally.

w

The fitted weights of the readout.

The penalty is chosen by nested cross-validation: pick the penalty on inner folds, score on outer folds, so the score never sees the data used to tune it. Think of a practice exam and a sealed final. The paper uses 5 outer and 3 inner folds.

The errors, as MAE in mg/dL: OpenSLA-B misses the next 2 hours' mean by 24.0, the minimum by 22.3 and the maximum by 28.0. Its slope error is 0.39 mg/dL per minute. OpenSLA-H scores 24.0, 22.5 and 27.7, with the same slope error. The signal readers score 26.5 and 26.9 on the mean. A glucose foundation model's own zero-shot forecast scores 29.7, and the other glucose model 33.2. The slope errors are close for almost everyone: OpenSLA and both signal readers all round to 0.39.

The paper's own caveat deserves quoting: "These results concern a trained readout of frozen representations, not zero-shot generation of future glucose values."

worked exampleThe unseen antibiotic (balanced accuracy)
  OpenSLA − best chat model         71.4 − 61.9      = 9.5 points
The new hospital (balanced accuracy)
  act or wait, B over best reader   74.40 − 69.30    = 5.10
  act or wait, H over best reader   73.00 − 69.30    = 3.70
  what kind, H over best reader     58.49 − 54.33    = 4.16
  windows per person                1,000 ÷ 87       ≈ 11.5
Ridge by hand (illustrative, one feature)
  x = 1, 2, 3    y = 2, 4, 7
  XᵀX = 1 + 4 + 9 = 14      Xᵀy = 2 + 8 + 21 = 31
  λ = 0          w = 31 ÷ 14                     = 2.214
  λ = 1          w = 31 ÷ (14 + 1)               = 2.067   (the penalty pulls w toward 0)
Future glucose, next 2 hours' mean (MAE, mg/dL)
  best signal reader − OpenSLA      26.5 − 24.0      = 2.5 mg/dL smaller

A new drug that never ranked

What the pharmacist sees. She adds "IMI-CIL 500" to the list of names. It never ranks, not even on windows where an antibiotic was clearly coming.

What you measure. Look at the descriptor's nearest names. It sits far from every antibiotic, because its tokens mean little to the language model. It is an abbreviation and a number, not words.

What you change. Write descriptors the way training wrote them: "category: name", as in the paper's "antibiotic: Imipenem/Cilastatin". The story is illustrative; the descriptor format is the paper's.

Keep the limits in view. The unseen-action test uses one drug, inside a kind the model had already seen; a brand-new kind of action is untested. The new-hospital set is 87 people. And the continuous structure is shown for one kind of action, insulin.

Imagine a trial run on a ward. One window in ten is followed by an action. The model's act-or-wait call is right on 72.5% of the action windows and on 72.5% of the quiet ones. A nurse's pager goes off. How likely is it that this alarm is real?

Most people guess about 72.5%. The true answer is about 23%. In plain words: a score measured on a half-and-half test set says little about how many false alarms a real ward would get. And this model learned what people recorded doing, which is not the same as what was right. This chapter works through both.

Counting alarms on a real ward

Two new words. Prevalence is the share of windows that really need action, like how common a disease is in a town. Precision is the share of alarms that are real, like how often the smoke alarm means fire.

Precision is easiest to see by counting, not by formula. Take 1,000 windows on a ward where 10% need action. That is 100 real action windows and 900 quiet ones. The model catches 72.5% of the 100: 72.5 alarms that are real. It also raises a false alarm on 27.5% of the 900 quiet windows: 247.5 false alarms. So the pager goes off 72.5 + 247.5 = 320 times, and only 72.5 of those are real. That is 22.7%.

Notice what happened. The model was the same; the ward changed. A rare event means a huge pool of quiet windows, and even a small false-alarm rate on a huge pool makes many false alarms.

Written as a formula, with s the sensitivity, t the specificity and π the prevalence:

precision

Of the alarms, the share that are real.

s

Sensitivity: the share of real actions the model catches.

t

Specificity: the share of quiet windows it leaves alone. 1 − t is its false-alarm rate.

π

Prevalence: the share of windows that really need action.

Derive it in four steps, the same way we counted.

1. Split the windows. Out of N windows, real actions = πN, and quiet windows = (1 − π)N.
2. Count the alarms. Caught, TP = s · πN. False alarms, FP = (1 − t) · (1 − π)N.
3. Take the share. Precision = TP / (TP + FP) = sπ / (sπ + (1 − t)(1 − π)). N cancels.
4. Check the balanced case. At π = 0.5 with s = t, precision = s: exactly the balanced test's score.

That last step is the link to the paper. Its test windows are 50/50 on necessity, so its balanced accuracy is also the precision you would see at 50% prevalence, when sensitivity equals specificity. Real wards are rarely 50/50. Slide the prevalence down and watch the pager fill with false alarms.

Two lessons come out of the device. The same model's precision collapses as actions get rarer: 72.5% at half and half, 22.7% at one in ten, about 5% at one in fifty. And at low prevalence, raising specificity helps precision far more than raising sensitivity does, because the quiet windows are where the false alarms come from. Unlock the two rates and try it.

When most alarms are false, people learn to ignore them, the real ones included. That is alarm fatigue, the boy who cried wolf, and it is a real hazard on wards. A model that floods a pager is worse than no model.

worked example1,000 windows, sensitivity = specificity = 0.725 (an even split of 72.5, illustrative)
  π = 0.50   real 500 · caught 362.5 · false alarms 0.275 × 500 = 137.5
             alarms 500 · precision 362.5 ÷ 500                    = 72.5%
  π = 0.10   real 100 · caught 72.5  · false alarms 0.275 × 900 = 247.5
             alarms 320 · precision 72.5 ÷ 320                     = 22.7%   missed 27.5
  π = 0.02   real 20  · caught 14.5  · false alarms 0.275 × 980 = 269.5
             alarms 284 · precision 14.5 ÷ 284                     = 5.1%

What the labels really are

The second caution is about what the model learned to copy. Its labels say "an action was recorded in the next few minutes" (Chapter 1). That is recorded practice: what clinicians actually did and wrote down, right or wrong. It is like copying the class's homework, mistakes included.

A model trained on recorded practice copies a hospital's habits, and with them its gaps and its biases. If a ward tends to give a drug late, or rarely tests one group of patients, the model learns that pattern as if it were the right one.

The captions carry the same caution. The paper describes them as evidence associations, not causal explanations or treatment recommendations. A fluent "why" is not a medical reason. It lists findings that fit the predicted action (Chapter 6).

The paper's own limits

The paper is explicit about what would come before any real use. Deployment would require prospective validation: testing on new patients as they arrive, before anyone relies on it. Think of a road test, not a simulator. It would also need a safety evaluation and appropriate regulatory review.

It names two design limits. It trains a separate model for each setting, partly because their action lists were defined separately. And its experiments assume a fixed set of sensors and channels, while real devices vary in what they record and when.

Three more facts from earlier chapters belong in the same list. The new-hospital test is 87 people. The weights were announced, but on 11 October 2026 they could not be downloaded. And each ± covers which patients were tested, not training reruns.

A trial that worked in the paper

What the ward sees. A trial switches the model on. After a week, staff stop answering its alerts.

What you measure. Compute precision at the ward's own prevalence from the logged alerts. About 1 alert in 4 is real. The model is behaving exactly as its test score promised; the ward is simply not 50/50.

What you change. Set the threshold for this ward's prevalence and for its cost of a missed action against a false alarm. Then run a prospective study before anyone relies on it. The trial is illustrative; the arithmetic is the chapter's.

What would change this picture? Publishing sensitivity and specificity separately, since the paper reports only their average. Calibrated probabilities, so a 0.7 means a 70% chance. And prospective trials on real wards.

You can now read this paper end to end and explain every design choice in it. Here is the whole of it on one page: the summary, the numbers, a checklist, the decision path as code, and where it connects.

The one-paragraph summary

Sensor models describe the body; they stop before the decision. OpenSLA adds the person's story and each action's name as text, so one model per setting can do three jobs. It decides on a three-rung ladder: act or wait, what kind, which one. It describes the window with numbers you can check. And it points to the evidence behind the decision. It made the best act-or-wait call in all six groups of patients and lost three specific cells. Its captions carry numbers that can be read back and scored. Writing sentences during training improved the decision itself. It is a research preprint, not a clinical tool.

The numbers that matter

The checklist

If you build a model like this, or judge one, these are the decisions this paper got right, in the order you meet them.

1. Fix the decision time, the history window and the target interval for each setting first.
2. Build quiet windows that cannot be guessed by a shortcut: match their history, and exclude recent actions.
3. Split by person, never by window.
4. Build captions by rule from extracted facts. Declare low-quality channels instead of describing them.
5. Feed sensors with a mask for missing minutes. Never fill them with zeros.
6. Compress long sensor inputs with learned queries. Skip compression when inputs are already short.
7. Read all three rungs from one hidden state. Write the gate rule and the fallback down.
8. Score names by matching their text, so new actions need no new weights.
9. Report balanced accuracy and AUROC, macro-averaged, with thresholds from validation and bootstrap intervals over people.
10. Check caption numbers by extraction and MAE. Never trust a wording score alone.
11. Before any ward, compute precision at that ward's prevalence, and validate on new patients first.

The decision path as code

Here is the whole decision path from h to the output record, from scratch in numpy. It mirrors the decision rules of the released code's prediction step; the caption is generated separately. The comment block at the top restates the input contract from Chapter 3.

pythonimport numpy as np

# The input contract of the released code, for one clinical window (B = 1 sample).
#   waveform               [B, M, C, T]  M = 30 one-minute slots, C channels, T = 7500 samples (60 s at 125 Hz)
#   waveform_channel_mask  [B, M, C]     True where that channel really recorded in that minute
#   waveform_modality_ids  [B, C]        which named channel each column is ("II", "PLETH", "RESP", ...)
#   numeric events         [B, E] each:  values, rel_time_min, measure_ids, source_ids, event_mask
#   input_text             one string:   the person's context and the question

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def opensla_decide(h, heads, label_index, need_thr=0.5, cat_thr=0.5, max_cats=5, k=3):
    """h: the language model's hidden state at the last prompt position, shape [2048].
    Mirrors the decision rules of the released predict(); the caption is generated separately."""
    p_need = float(sigmoid(heads["w_need"] @ h + heads["b_need"]))       # rung 1: act or not
    p_cat = sigmoid(heads["W_cat"] @ h + heads["b_cat"])                  # rung 2: one switch per category
    needed = p_need >= need_thr
    chosen = [g for g in range(len(p_cat)) if needed and p_cat[g] >= cat_thr]
    if needed and not chosen:
        chosen = [int(np.argmax(p_cat))]                                  # gate open, none passed: keep the best
    chosen = sorted(chosen, key=lambda g: -p_cat[g])[:max_cats]           # most likely first, at most 5
    q = heads["query"](h)
    q = q / np.linalg.norm(q)                                             # rung 3: project h, then match names
    fine = {}
    for g in chosen:
        names, vecs = label_index[g]                                      # vecs: projected name vectors [n, 2048]
        cos = vecs @ q / np.linalg.norm(vecs, axis=1)
        top = np.argsort(-cos)[:k]
        fine[heads["cat_names"][g]] = [{"label": names[j], "cosine_similarity": round(float(cos[j]), 3)} for j in top]
    return {"necessity_probability": round(p_need, 3), "action_needed": bool(needed),
            "category_probabilities": dict(zip(heads["cat_names"], np.round(p_cat, 3).tolist())),
            "selected_categories": [heads["cat_names"][g] for g in chosen], "fine_labels": fine}

if __name__ == "__main__":                                                # a smoke test with random toy weights
    rng = np.random.default_rng(0); d = 2048
    heads = {"w_need": rng.normal(0, .02, d), "b_need": 0.0, "W_cat": rng.normal(0, .02, (3, d)), "b_cat": np.zeros(3),
             "query": lambda h: h, "cat_names": ["antibiotics", "antiemetic", "fluids"]}
    label_index = [(["meropenem", "cefepime"], rng.normal(size=(2, d))), (["ondansetron"], rng.normal(size=(1, d))),
                   (["saline bolus"], rng.normal(size=(1, d)))]
    print(opensla_decide(rng.normal(size=d), heads, label_index))
# {'necessity_probability': 0.156, 'action_needed': False, 'category_probabilities':
#  {'antibiotics': 0.754, 'antiemetic': 0.641, 'fluids': 0.756}, 'selected_categories': [], 'fine_labels': {}}

Read it in four blocks. The first two lines of the function are the first two rungs: one sigmoid for the gate, one per kind. The next four lines are the rules from Chapter 5. Kinds are chosen only if the gate is open; if it is open and none passes, the single best is kept; then they are sorted and capped at 5.

The loop is the third rung: normalise the window's query, take each name's cosine with it, and keep the top 3. The return statement builds the same record the device in Chapter 5 showed, field for field.

Now look at the smoke test's output. All three kinds score above 0.5: 0.754, 0.641 and 0.756. Yet nothing is selected, because the gate gave 0.156. That is the gate rule once more, and it is the most important line in the ladder: below the gate, nothing is asked.

The paper

Yuekai Xu and Zitao Shuai (co-first authors) and Yuzhe Yang (corresponding author), University of California, Los Angeles. "Sensor-Language-Action Models." arXiv 2610.08244, posted 6 October 2026, a preprint with no venue yet. The code is at github.com/yang-ai-lab/OpenSLA under the MIT licence, released on 7 October 2026. An OpenSLA-H clinical checkpoint, built on Qwen3.5-2B, is announced at huggingface.co/yang-ai-lab/OpenSLA; on 11 October 2026 the repository held only its README. The work was funded by NIH R21-EB038753, the Amazon Research Awards, an NVIDIA Academic Grant and the Biswas Foundation.

What it builds on, and what came after

Signal foundation models and sensor-language models. The rivals Chronos-2, Moirai and GluFormer learn general signal features; SensorLM, SleepLM and OpenTSLM learn to match or describe signals in words. All of them stop at understanding. OpenSLA's wave encoder is adapted from the same group's sleep foundation model, OSF.

Vision-language-action models. Robot models that output actions from images and instructions are the action-only recipe that OpenSLA beat in its own comparison (Chapter 10). LLaVA and Flamingo. The projector design and the gated cross-attention design it compared against LoRA come from these two vision-language models.

What came after. On 11 October 2026 the paper is five days old, and nothing has built on it yet. One next step follows from its own limits: the paper trains one model per setting because the action lists were defined separately, so a single model across all settings is the obvious target. That is our reading, not the paper's claim.

Where it sits in the catalog

This lesson joins no Path. These lessons connect to it:

• Time-series forecasting: predicting a signal's future from its past, the job Chapter 11's glucose readout is not.
• Multimodal foundation models: how images and other inputs enter language models, the family Chapter 3 extends to body signals.
• Audio language models: another sensor, sound, fed to a language model with the same prefix-or-adapter choices.
• Motion embeddings for activity recognition: wearable motion signals from labels to text, the perception side this paper starts from.
• Passive heart-rate monitoring: getting a heart rate from sensors, a pure perception task.
• LLaVA-1.5: the projector design that scored 51.5 here (Chapter 10).
• DuplexSLA: a different "SLA", speech, language and action, for voice agents. The same three-part idea with a different S.

For the prerequisites in more depth: LoRA (Chapter 6), the Perceiver and learned queries (Chapter 4), evaluation statistics and the bootstrap (Chapter 7), vision-language-action models and contrastive alignment (Chapter 10).

Go back to the monitor at the top and ask it again. You now know what each answer is made of: the window and its "now" (Chapter 1), the sentences it learned from (2), the tokens and the memory (3, 4), the ladder (5), the gated writer and its evidence (6), and what the score bars do and do not promise (7, 12). Then teach it: Present walks you through saying it aloud, and Teach gives you a whiteboard. Back to the monitor.