Surgical Tool Recognition for OR Nurse Training

The Challenge

New nurses and OR staff must learn large sets of surgical instruments, prepare realistic trays, distinguish near-lookalikes, and hand the requested tool to the surgeon quickly under pressure. A wrong choice can slow the case, create stress for a new nurse, and frustrate the surgical team.

Static flashcards do not reproduce the visual conditions of a real tray: polished metal reflections, clutter, rotation, overlapping instruments, partial occlusion, unfamiliar viewing angles, and tools that differ only in jaw pattern, curvature, length, or locking mechanism.

A related robotics project at Sheba needs many of the same recognition scenarios for an industrial arm with a medical gripper. The human-training challenge stands alone, while generated tray layouts could optionally become synthetic test cases for robot vision and retrieval.

Reference tray photo showing a set of similar-looking surgical instruments — Reference tray photo: the visual-recognition challenge is partly about subtle differences between similar instruments in a crowded tray.

The Big Idea

Use the object-search and matching loop of Triple Match 3D as one reference, then redesign it around real tray preparation and tool handoff. Teams may choose a completely different game loop if it trains the same skill more effectively.

Triple Match 3D gameplay showing a crowded field of objects to search and match

What a Jam Prototype Could Be

A 3D matching or search game using instrument models: find three of the same tool, sort a procedure tray, or pick and hand off the requested instrument.
Three to five procedure-layout modes based on common surgeries, such as lumpectomy, hand surgery, and plastic surgery, where the player prepares the correct layout or identifies the surgery from the tray they see.
Escalating difficulty with clutter, reflections, partial occlusion, rotation, and visually similar tools.
Training contrasts such as Kelly clamp vs. Crile clamp or other lookalike instruments chosen by clinical reviewers.
Immediate feedback explaining why the selected tool was right or wrong.
Optional synthetic tray generator for robot-vision testing, if the team has computer-vision interest.
Scenario modes for learning names, discriminating lookalikes, memorizing a procedure-specific tray, responding to a spoken request, and recovering after an incorrect choice.

Clinician-Provided Reference Examples

Matan shared early reference examples for lumpectomy and hand-surgery tray preparation. These are not a required design direction, but they show the kind of real layout knowledge the game could teach: recognizing procedure-specific sets, preparing the room correctly, and choosing the right supporting materials before the case begins.

Lumpectomy tray preparation reference example — Lumpectomy setup reference example.

Second lumpectomy tray preparation reference example — Additional lumpectomy reference example.

Hand surgery tray preparation reference example — Hand surgery setup reference example.

Include supporting supplies, not only instruments: sponges, gauze, sterile towels, sutures, covers, bowls, motors, tables, and procedure-specific sets.
Train suture categories, such as absorbable Monocryl, non-absorbable Nylon, and monofilament sutures.
Practice matching room setup to the case, for example hand surgery kit, finger set, hand accessory set, table cover, wash basin, Stryker motor, and hand table for a basic hand surgery.
Include scalpel blade decisions, for example #10 or #15 for lumpectomy, #15 for hand surgery, and #11 for laparoscopic surgery.

Design Questions

Which instruments are most important for new OR staff to learn first?
Which three to five common procedures should anchor the first tray-layout scenarios?
What mistakes happen under pressure?
Could players learn both directions: build the right layout for a named procedure, and identify a procedure from the layout shown?
How can the game train subtle visual recognition without becoming a flashcard app?
What assets can safely be used: approved training photos, synthetic models, or non-identifiable reference images?
Should performance prioritize accuracy first, then speed, so pressure does not teach guessing?

What This Is Not

Not a real-time intraoperative tool-selection system.
Not a substitute for formal OR training or supervision.
Not a place to use identifiable OR footage.

Why This Matters

The right prototype could help staff practice before the stressful moment, while also giving robot-vision teams a structured way to think about tray complexity.