01 · Behavior
What might it do?
Compare possible actions and their probabilities. Revise the distribution to express a preferred choice.
Behavior edits guide the ActorEditable explanations for reinforcement learning
Editable XRL
Edit an agent’s actions, predictions, and values.
CoChoice connects these parts of a decision in an editable Markov tree. People can reconsider their own expectations, correct the agent’s reasoning, and retain those corrections as it continues learning.
Collaboration · Transportation · Medicine · Cognitive modeling
The CoChoice workflowRead · Revise · Reinforce
01 / Method
Where does the disagreement come from?
A choice depends on the action, the consequences the agent expects, and the value it assigns to those consequences. CoChoice makes all three inspectable and editable.
02 · Belief
What does it expect?
Inspect predicted consequences, including a partner’s response. Correct an expectation that conflicts with your knowledge.
Belief edits guide the World Model03 · Benefit
What does it value?
Compare the values assigned to outcomes. Revise an evaluation to express what makes an outcome desirable.
Benefit edits guide the CriticThe tree provides a shared basis for reconsideration and correction. Edits become supervision for the corresponding model components; further reinforcement learning retains that guidance alongside the task objective.
02 / Guided demonstration
Walk through a role mismatch.
One cooking decision, seen through the agent’s choices, expectations, and values.
01 / The mismatch
You take the onions. The AI should serve.
You are the green chef. You want the blue AI to handle dishes and soup. But it favors collecting onions, with a 68% probability—so both partners want the same job.
What participants observed
After Revise“Taking the plate was expected, but then not going to take the soup was not expected … they just took the plate.”
Revise11 · manuscript §6.2.2
After Reinforce“It picked up the dish, picked up the soup, and served soup.”
Reinforce13 · manuscript §6.2.3
Your intended roleCollect onions
AI’s current preference68% onion pickup
The explanation makes the competing intentions visible before you make a correction.
1 of 5
Illustration based on Figures 3–4. The edit is a training target; the final step presents separate observations from the study (§6.2).
03 / Collaboration study
From accommodating the agent to coordinating with it.
In a randomized, between-subjects study, people collaborated with an AI in Overcooked. Three conditions compared reading explanations, revising them, and retaining edits during continued learning.
39participants
across three conditions
across three conditions
| Measure | Read | Revise | Reinforce |
|---|---|---|---|
| Human understanding of AIAccuracy in predicting the agent’s intention | 91.9% | 49.6% | 93.8% |
| Human role fulfillmentMaintaining the assigned onion-collection role | 19.7% | 75.8% | 95.6% |
| AI understanding of humanAccuracy in predicting the human’s intention | 77.1% | 73.4% | 87.8% |
Post-intervention group means; higher values indicate greater understanding or role fulfillment. These results describe the Overcooked study. Source: manuscript §6.3 and Figure 5.
04 / Beyond collaboration
Inspecting dispatch, treatment, and attention.
Six stakeholders explored the same action–consequence–value structure in three domain probes. Their corrections show how different kinds of expertise can guide a decision.
Transportation / Ride-hailing repositioning
Will the driver go where the system recommends?
A recommendation’s intended effect on supply–demand balance depends on whether the driver accepts it and arrives in the target region. The probe connects recommendations to the responses that determine their effect.
- Behavior
- Recommended destinations
- Belief
- Predicted driver destinations
- Benefit
- Supply–demand balance and preference satisfaction
A targeted correction
A stakeholder proposed correcting the predicted driver-response distribution when it conflicted with observed behavior, then revisiting the dispatch recommendation using that corrected belief.
What further adaptation needs
Driver preferences depend on context such as traffic and expected income. Stakeholders wanted these changing circumstances and potential hazards to be represented explicitly.
Medicine / Retrospective sepsis-treatment probe
Which predicted changes justify a treatment?
Different recommendations may reflect different expectations about the patient’s response. The probe connects treatment choices to predicted physiological trajectories over 24 hours and estimated mortality risk.
- Behavior
- Treatment options and their selection probabilities
- Belief
- 24-hour physiological trajectories and their probabilities
- Benefit
- Estimated mortality risk
A targeted correction
One participant increased the preference for fluids and vasopressor support over observation while retaining forecasts they considered plausible. Another wanted to correct physiological trends without specifying exact values.
What further adaptation needs
Risk estimates need interpretable physiological evidence. Subsequent choices should incorporate the patient’s observed response, with forecasts expressed at a clinically meaningful level of precision.
Cognitive modeling / Hierarchical reading
Which reading decision led attention astray?
A hierarchical RL model decides what to read at the text, sentence, and word levels. The probe exposes how those choices lead to particular fixations and the information the model expects to acquire.
- Behavior
- Choices at the text, sentence, and word levels
- Belief
- Information expected along the selected route
- Benefit
- Controller utility, with higher values preferred
A targeted correction
A participant traced unexpected emphasis on “eco-friendly” to the sentence reader. They proposed raising “skip next word” to about 50%, reducing “read next word,” and revising the associated utility.
What further adaptation needs
A correction at one controller expresses a hypothesis about the reading process. Extending it requires behavioral evidence and cognitive theory about how that change should affect later choices.
Evidence: six stakeholders, two per domain, in 45–60 minute interviews. These formative probes elicited corrections and design requirements; performance improvements from retraining were not evaluated. Manuscript §7, Figures 8–10.
About the research