Soft Gripper Deformation Annotation: Training AI for Compliant Manipulation of Fragile Objects

Modern robotics is moving towards tasks that require interaction with delicate and deformable objects, such as food products, medical materials, and fragile household items. In these conditions, traditional rigid grippers are ineffective due to the risk of damaging objects and their limited adaptability to their shapes.

Soft robotic grippers offer a fundamentally different approach to manipulation, based on compliant interaction with the environment. Due to their deformable structure, they can passively adapt to the shape of the object, reducing contact stresses and increasing grip safety. However, such flexibility creates challenges for modeling and control, as the gripper's behavior is highly nonlinear and depends on many external factors.

Problem statement

The goal of this research is to develop an approach for training artificial intelligence models to control soft robotic grippers in fragile-object grasping scenarios, where it is critical to ensure safe, stable interaction with fragile or deformable objects.

The main difficulty is that a soft gripper's behavior is essentially compliant: its shape changes under external forces, and contact with the object is unevenly distributed. This makes it impossible to use classical rigid kinematic models and requires specialized data and methods to represent the system's state.

Within the framework of the problem, it is necessary to:

• form a structured soft gripper dataset that reflects the interaction of the gripper with objects of different shapes and stiffness;
• perform deformation annotation to describe spatial changes in the shape of the soft gripper during contact;
• perform contact area labeling to identify the interaction zones between the gripper and the object;
• integrate pressure map data to display the pressure distribution in the contact area;
• develop an approach to compliant manipulation training that uses multimodal data to train control models.

The ultimate goal is to build a model that can predict and adapt the behavior of a soft gripper during gripping, minimizing the risk of damage to the object and ensuring stable contact during manipulation. This should increase the efficiency of robotic gripping systems for tasks involving delicate, deformable objects.

Overview of approaches

One of the key areas is the development of soft grippers, which, due to their compliant structure, allow for reducing local loads on the object. Tactile sensors and cameras are widely used in such systems to obtain information about contact; however, interpreting this data remains a difficult task due to the high nonlinearity of deformations.

In training models for controlling such systems, machine learning-based approaches play a significant role, particularly deep neural networks, which can handle multimodal input data. At the same time, the effectiveness of such methods largely depends on the availability of high-quality datasets, particularly soft-gripper datasets that provide synchronized information on the gripper state and contact conditions.

Existing works in the field of tactile sensing and vision-based manipulation are often limited to either visual data or local pressure measurements, which makes it difficult to build a complete interaction model.

A separate area of ​​research is the study of deformations of soft structures, where computer vision and 3D reconstruction methods are used to analyze the shape of a soft gripper during contact. However, deformation annotation remains a complex process, as it requires precise matching of spatial changes with the temporal dynamics of interaction.

Problems of a lack of structured data

Most existing approaches collect data in a fragmented manner: visual observations, tactile measurements, or pressure signals are stored separately, but their full synchronization is rarely ensured. This makes it difficult to build models that can correctly interpret multimodal signals in real time.

The lack of agreed-upon standards for deformation annotation is particularly critical, as different studies use incompatible approaches to describe soft-gripper deformation, ranging from simple heuristic metrics to complex 3D mesh representations. This leads to results from different works being difficult to compare, and to models not generalizing well across conditions.

A similar problem is observed in contact area labeling, where the lack of a single methodology for determining the contact area leads to significant discrepancies in labeling across the same gripping scenarios. As a result, models can be trained on noisy or inconsistent labels, thereby reducing training stability.

Another significant problem is the lack of high-quality pressure map data obtained under real-world conditions. In many works, such data are either simulated or collected with low spatial resolution, which limits the ability to accurately model the force distribution in the contact zone.

Structured issues in building a soft gripper dataset

Deformation annotation pipeline

The deformation annotation pipeline in the soft gripper dataset is designed to provide a formal description of the shape changes of a soft gripper during fragile-object grasping tasks. Since a soft gripper exhibits continuous nonlinear deformations, the key challenge is to present these changes in a structured form suitable for subsequent compliant manipulation training.

In general, deformation can be represented in different ways. The simplest approach is the keypoint-based description, where control points are set on the gripper surface, and the deformation is defined as their displacement relative to the baseline state. The mesh-based approach is more accurate, in which the surface is modeled as a mesh and the movements of its vertices are tracked. The most detailed is the dense representation, which is based on a pixel-by-pixel or field-by-field description of deformations, for example, through displacement vectors similar to optical flow.

The annotation process is usually hybrid: part of the data is specified manually to ensure correct initialization and quality control, while most of the information is generated automatically. This is done using computer vision techniques, including feature point tracking, marker-based tracking, and depth reconstruction from RGB-D sensors. This allows for scaling the creation of a soft gripper dataset without excessive manual intervention.

In a typical pipeline, a reference undeformed state of the gripper is first determined, followed by multimodal data collection, including visual information and pressure map data. This is followed by tracking shape changes over time and calculating deformation vectors relative to the baseline state. The resulting data is synchronized with contact information, including contact area labeling, enabling alignment of geometric changes with physical interactions. The final stage involves noise filtering, trajectory smoothing, and data normalization for stable model training.

Contact modeling and pressure representation

Contact modeling and pressure representation are critical steps in building a soft gripper dataset, as these data define the physical interaction between the gripper and the object in fragile-object grasping tasks. Unlike rigid manipulators, in a soft gripper, the contact is distributed, continuous, and strongly dependent on the material's deformation, which requires a specialized approach to its description.

The contact area is usually defined as the  ​​intersection of the gripper surface and the object, where non-zero pressure or mechanical interaction occurs. In practical systems, this area can be obtained through image segmentation, marker tracking, or tactile sensor data. The result is a contact mask, which is often consistent with contact-area labeling and serves as a spatial reference for further analysis.

Pressure map data is formed by discretizing the force distribution in the contact area. Depending on the type of sensor, this can be a grid of pressure values ​​obtained from tactile matrices or a reconstructed pressure map based on material deformation. In more complex systems, interpolation between sensor elements is used to obtain a continuous pressure field, which allows a better representation of the real physics of the interaction.

To train models for compliant manipulation, various features are extracted from this representation. These include the area and shape of the contact area, maximum and average pressure values, force distribution gradients, and spatial correlations between pressure and deformation. Such features enable the model to better assess grip stability and predict the risk of slipping or damage to the object.

Compliant manipulation training method

FAQ

What is the main goal of a soft gripper dataset in robotic manipulation?

A soft gripper dataset aims to provide structured multimodal data for learning how deformable grippers interact with objects. It supports compliant manipulation training by capturing visual, tactile, and force-based signals during interaction.

Why is deformation annotation important in soft robotics?

Deformation annotation is essential because it describes how the soft gripper changes shape during contact. This information enables models to understand non-rigid behavior in fragile-object grasping scenarios.

What role does contact area labeling play in the dataset?

Contact area labeling defines the exact regions where the gripper interacts with the object. It improves the accuracy of learning physical interaction patterns in compliant manipulation training.

How is pressure map data used in manipulation learning?

Pressure map data represents the distribution of force across the contact surface. It helps models estimate grasp stability and prevent damage during grasping fragile objects.

Why is multimodal data important for soft grippers?

Multimodal data combines visual, tactile, and force information into a unified representation. This is crucial for building robust policies in a soft gripper dataset.

What challenges arise without structured datasets?

Without a well-organized soft gripper dataset, models struggle with generalization and stability. Missing or inconsistent deformation annotations and pressure data degrade performance on real-world tasks.

How does deformation annotation interact with contact area labeling?

Deformation annotation and contact area labeling are strongly correlated because deformation often occurs at contact points. Together, they provide a comprehensive view of the interaction geometry in fragile-object grasping.

What is the purpose of compliant manipulation training?

Compliant manipulation training aims to teach models to safely adapt to physical interactions. It ensures that soft grippers can handle uncertainty and avoid damaging objects.

What is the role of pressure maps in control policies?

Pressure map data helps control policies, estimate force distribution, and adjust actions accordingly. This improves grasp stability and reduces failure in dynamic environments.

Why is soft gripper research important for robotics?

Research on the design of soft gripper datasets improves robotic adaptability in real-world environments. It enables safer, more reliable grasping of fragile objects across diverse applications such as healthcare and food handling.