Analogical Reasoning Annotation: Training LLMs to Transfer Knowledge Across Domains

For AI LLMs, cross-domain knowledge transfer is still a challenge. The superficial correlation of words prevents AI from seeing hidden geometric and logical connections. To train LLMs to build complex abstractions, developers are moving from simple text parsing to creating specialized datasets.

In this article, we will look at how to build an analogical reasoning dataset, which tools to use for structural coding, and how to validate AI's ability to transfer knowledge across domains.

Quick Take

• Most LLMs find superficial text matches, but are blind to deep pattern-matching detection.
• Training models require detailed relational mapping data that capture roles, functions, and cause-and-effect relationships.
• The domain transfer labeling process requires labeling metaphorical and abstract models.
• To test whether the model "understands" the analogy and does not simply remember it from the training data, synthetic and abstract benchmarks with high variability are used.

What is Structural Analogy Annotation, and why is it difficult?

Currently, LLMs are trained to predict the next token. This works for factual answers, but is powerless when the model needs to apply a principle from biology to solve an architectural or engineering problem (biomimicry).

Structural analogy annotation is a data markup process in which textual or formal descriptions are encoded as graphs of relationships. The markup focuses on structural isomorphism, preserving the system of relationships.

The markup does not label the object itself, but its functional role in the system. This allows the model to understand that the deep logic of relationships and cause-and-effect relationships between objects in one domain are structurally equivalent to the system of interactions between completely different objects in the target domain.

Anatomy of Creating Relational Mapping Data

Building quality relational mapping data consists of three stages.

The goal is to transform unstructured text into a clear schema of entities and predicates.

Step 1: Source Domain Decomposition

The text is broken down into its basic elements:

• Entities: E₁, E₂,...
• Predicates: P(E)
• Higher-order Relations: R(E₁, E₂) or CAUSES(R₁, R₂)

Step 2: Target Mapping

The annotator searches for or creates a target domain that holds the logical form R(E₁, E₂) R(E₃, E₄), but the entities E₃, E₄ themselves belong to a completely different field (e.g., economics vs. physics).

Step 3: Pattern Matching Detection

Add metadata that tells the model the type of analogy:

• By functionality (e.g., heart : pump)
• By causality (e.g., drought : crop failure :: inflation : decline in purchasing power)
• By spatial-structural systematicity

Domain Transfer Labeling Practice for Multimodal and Text Models

The domain transfer labeling process involves creating complex scenarios where AI developers give the model a task from the source domain and require it to generate a solution for the target domain.

Thanks to this markup, the model learns to operate on abstract task graphs, rather than simply paraphrasing text.

How to Build a Reasoning Evaluation Set

To test whether a model is capable of analogous thinking, a specialized reasoning evaluation set is needed. Regular test datasets are not suitable because the model tends to memorize well-known Internet metaphors (e.g., "life is a journey").

Requirements for the evaluation dataset:

1. Novel Domains. Use of fictional worlds or concepts (e.g., fantastic physical laws) to make memorization impossible.
2. Counterfactual scenarios. Change one detail structurally (e.g., instead of attraction, repulsion) to test whether the model notices this.
3. Distractor Items. Include answer options that have high lexical similarity but a completely different logical structure (a test for resistance to superficial hallucinations).

Markup Methodology and QA Workflow

Creating an analogical reasoning dataset requires annotators to engage in high-level abstract thinking and exert cognitive effort. Key stages of the analogy markup pipeline:

Inter-Annotator Agreement. Since the perception of metaphors and analogies is subjective, the same relational graph is evaluated by 3–5 independent experts. The final schema of the relational mapping data is accepted when the consistency index (for example, Cohen's k or Fleiss' k) exceeds 0.85.

Negative Transfer Prevention. The greatest danger in cross-domain knowledge transfer is when the analogy "breaks" at a certain step. Annotators are required to explicitly mark the boundaries of the analogy's applicability (Analogy Boundary Flags).

• Example: If the structural model conveys the interaction between objects but does not take into account the time factor, the time_dimension_mismatch flag is set. This teaches LLM not to transfer logic beyond the allowed context unthinkingly.

Synthetic graph augmentation. To increase the scale of the dataset, formal graph operations are used:

• Perturbation. Replacing individual graph nodes with synonymous concepts while preserving relational relationships.
• Inversion. Changing the direction of cause-and-effect vectors to create negative examples for training pattern-matching detection.

FAQ

What is the difference between simple text classification and analogical reasoning annotation?

Classification assigns a text to a specific category (e.g., "Sports" or "Finance"). Analogical reasoning annotation detects and labels the deep logical structure (isomorphism) between two completely different texts from different categories.

Why is it difficult for LLM to build analogies on its own without a special dataset?

Transformer models are optimized for the probability of word coexistence (statistical context). They prefer superficial matches by keywords. Without special relational mapping data, the model misses the abstract cause-and-effect relationship.

What types of analogies are the most difficult to label for training models?

The most difficult are cross-domain structural analogies with many levels of nesting – for example, comparing mathematical category theory with molecular biology or game theory in economics.