Human-Robot Object Handover Annotation: Labeling Timing, Grip Release, and Gaze Cues

The transfer of objects between humans and robots is an important component of effective and natural interaction in robotic systems. Successful execution of this process requires the robot to correctly interpret human behavioral signals, including movement timing, the moment of loosening or releasing the grip, and gaze direction.

Clear and consistent labeling of the timing of interaction, release of the grip, and gaze signals allows for a detailed description of the dynamics of the object transfer process and creates a reliable basis for the development of robots that can better understand human intentions, respond to human actions in a timely manner, and ensure safe and natural interaction.

Description of the process of transferring an object between a human and a robot

Transferring an object between a human and a robot requires precise coordination of both participants' actions. During the interaction, not only are physical movements important, but behavioral signals are also important in determining readiness to transfer or accept the object.

A typical process of transferring an object includes the following stages:

• Initiation of transfer - one of the participants holds the object and begins moving it towards the other participant.
• Approaching the transfer area - the object is moved to a position convenient for its reception.
• Preparation for grasping - the recipient directs his hand towards the object and prepares to grasp it.
• Contact with the object - the recipient touches it and begins to hold it.
• Joint holding - for a short period, both participants can hold the object simultaneously.
• Loosening and releasing the grip - the participant transfers the object, reducing the grip force and fully releasing it.
• Completion of transfer - the object is fully under the control of the recipient.

During annotation, special attention is paid to three key groups of signals:

• Timing - the moments of start, contact, transfer, and end of interaction.
• Grip Release - the moment when the participant stops holding the object.
• Gaze Cues - the direction of gaze at the object, robot, person, or other area.

Labeling timing

During the transfer of an object, it is important to accurately determine when each stage of interaction begins and ends. To do this, key time points are marked in the video recording: the beginning of the hand's movement with the object, approaching the transfer zone, the first contact with the object, its joint holding, and the completion of the transfer.

Time marking is performed using video frames or timestamps. This allows you to trace the sequence of actions of the person and the robot and determine how long each stage takes.

In some cases, it is difficult to determine the exact moment of the event, for example, when the hands overlap or the required action is poorly visible in the frame. In this case, the video is viewed frame by frame, and ambiguous moments are marked separately.

Annotating the Grip Release Moment

The moment of grip release is one of the key stages of an object transfer. It shows when the object is no longer held by the person transferring it and has been completely transferred to the other participant. Correctly determining this moment is important for understanding how smooth and coordinated the transfer is.

When annotating, you need to find the frame in which the hand begins to loosen its grip and then completely loses contact with the object. These two moments can occur almost simultaneously, so if necessary, the video is viewed frame by frame. The main timestamp is the moment when the object is fully released.

Sometimes it is difficult to determine the exact moment of release due to overlapping hands, poor angle or rapid movement. In such cases, it is worth focusing not only on the position of the fingers but also on the object's subsequent movement. If the object remains under the recipient's control after release, the transfer can be considered complete.

Gaze Cues

Gaze is an important cue during object transfer because it can indicate where a person’s attention is focused at a given moment. A person may be looking at an object, a robot’s hand, the robot itself, or in another direction. Gaze changes often occur in conjunction with other actions and can help to better understand the participant’s intent.

Annotation determines where a person’s gaze is directed at different stages of the transfer. For example, before the action begins, attention may be focused on the work; during the transfer itself, it may be focused on the object or point of contact. After the interaction is complete, gaze direction may also change.

It is not always possible to determine the gaze direction precisely. This may be due to head position, video quality, camera angle, or the person’s eyes being partially closed. In such cases, it is better not to make assumptions, but to mark the gaze direction as uncertain. This approach helps to avoid errors and makes annotation more consistent.

Annotation rules and criteria

To ensure consistency, the same annotation rules should be applied to all video recordings. The main annotation criteria are presented in the table below.

Ambiguous cases and their resolution

FAQ

What does handover annotation involve in human-robot interaction?

Handover annotation involves labeling key events during the transfer of an object between a human and a robot. It helps identify when the handover starts, develops, and ends.

Why is grip release timing data important?

Grip release timing data indicates the moment the giver releases the object. This information helps evaluate whether the transfer is smooth, safe, and properly coordinated.

What is gaze cue labeling?

Gaze cue labeling identifies where a person is looking during an interaction. The gaze may be directed toward the object, the robot, the robot’s gripper, or another area.

What information does a reaching profile dataset contain?

A reaching profile dataset may include the beginning of a reaching movement, its duration, contact with the object, and the final hand position. This data helps study movement patterns during object handovers.

What is the purpose of social signal annotation?

Social signal annotation focuses on non-verbal behavior such as gaze, gestures, body movements, and interaction timing. These signals provide additional information about a person’s attention and intentions.

What indicates the beginning of a handover?

The beginning of a handover is usually marked by the first frame showing the object's intentional movement toward the receiver. The same criterion should be applied consistently across all recordings.

What should be done when a grip release event is ambiguous?

If the exact moment of release is unclear, the video should be reviewed frame by frame. If the event still cannot be identified reliably, it should be marked as uncertain rather than guessed.

Why is consistent gaze cue labeling necessary?

Consistent gaze cue labeling makes it possible to compare behavior across different handover episodes. It also improves the reliability of data used for human-robot interaction training.

What role does annotated data play in human-robot interaction training?

Annotated data provides examples of human movements, gaze behavior, and object release timing. These patterns help robots recognize human intentions and respond at the appropriate moment.

What makes a handover annotation dataset reliable?

A reliable dataset requires clear annotation rules, consistent labels, and careful handling of uncertain cases. Agreement between annotators can also improve the overall quality of the dataset.