Skip to content

Turning Regular Models into Late Interaction Models

The LateWrapper class can turn any regular topic model into a late-interaction model. This might be more or less advantageous depending on the model.

Here's how LateWrapper works:

  1. It encodes documents at a token or window-level based on its parameters.
  2. It flattens the span embedding array, and feeds the this into the topic model, along with the token/window text.
  3. It unflattens the output of the topic model (doc_topic_matrix) into a ragged array, where you get topic importance for each span.

Let's see how this works in practice, and create different types of models and late interaction:

from sklearn.datasets import fetch_20newsgroups
from turftopic import Topeax
from turftopic.late import LateWrapper

corpus = fetch_20newsgroups(subset="all", categories=["alt.atheism"]).data

model = LateWrapper(
    Topeax(),
    window_size=50, # If we don't specify window size, it will use token-level embeddings
    step_size=40, # Since the step size is smaller than the window, we will get overlapping windows
)
doc_topic_matrix, offsets = model.fit_transform(corpus)
model.print_topics()
from sklearn.datasets import fetch_20newsgroups
from turftopic import SensTopic
from turftopic.late import LateWrapper

corpus = fetch_20newsgroups(subset="all", categories=["alt.atheism"]).data

model = LateWrapper(
    SensTopic(),
    span_extractor="sentence",
)
doc_topic_matrix, offsets = model.fit_transform(corpus)
model.print_topics()
from sklearn.datasets import fetch_20newsgroups
from turftopic import ClusteringTopicModel
from turftopic.late import LateWrapper

corpus = fetch_20newsgroups(subset="all", categories=["alt.atheism"]).data

model = LateWrapper(
    ClusteringTopicModel(n_reduce_to=10, feature_importance="centroid", reuction_method="smallest")),
)
doc_topic_matrix, offsets = model.fit_transform(corpus)
model.print_topics()
Topic ID Highest Ranking
0 morality, moral, morals, immoral, objective, behavior, instinctive, species, inherent, animals
1 matthew, luke, bible, text, passages, mormon, texts, translations, copy, john
2 atheism, agnostics, atheist, beliefs, belief, faith, contradictory, believers, contradictions, theists
3 punishment, cruel, abortion, penalty, death, constitution, homosexuality, painless, capital, punish
4 war, arms, invaded, gulf, hussein, civilians, military, kuwait, peace, sell
5 islam, islamic, muslim, qur, muslims, imams, rushdie, quran, koran, khomeini

The document-topic matrix, we created, is now a ragged array and contains document-topic proportions for each window in a document. Let's see what this means in practice for the first document in our corpus:

import pandas as pd

# We select document 0, then collect all information into a dataframe:
window_topic_matrix = doc_topic_matrix[0]
window_offs = offsets[0]
document = corpus[0]
# We extract the text for each window based on the offsets
window_text = [document[window_start: window_end] for window_start, window_end in window_offs]
df = pd.DataFrame(window_topic_matrix, index=window_text, columns=model.topic_names)
print(df)

                                                    0_morality_moral_morals_immoral  1_matthew_luke_bible_text  ...  4_war_arms_invaded_gulf  5_islam_islamic_muslim_qur
From: acooper@mac.cc.macalstr.edu (Turin Turamb...                         0.334267               1.287207e-13  ...             2.626869e-26                1.459101e-04
alester College\nLines: 55\n\nIn article <C5sA2...                         0.360400               8.898302e-14  ...             3.290858e-26                1.382718e-04
u (Mike Cobb) writes:\n> I guess I'm delving in...                         0.847002               5.002921e-22  ...             4.852574e-41                3.141366e-07
this you just have a spiral.  What\nwould then ...                         0.848413               5.819050e-22  ...             8.139559e-41                3.286224e-07
, even though this would hardly seem moral.  Fo...                         0.863685               1.272204e-21  ...             2.823941e-41                2.815930e-07
whatever helps this goal is\n"moral", whatever ...                         0.864913               1.584558e-21  ...             5.780971e-41                3.003952e-07
a "hyper-morality" to apply to just the methods...                         0.865558               1.919885e-21  ...             1.251694e-40                3.231265e-07
not doing something because it is\n> a personal...                         0.868360               2.951441e-21  ...             3.085662e-40                3.494368e-07
we only consider something moral or immoral if ...                         0.872827               5.444738e-21  ...             4.708349e-40                3.580695e-07
here we have a way to discriminate\nmorals.  I ...                         0.876951               1.021014e-20  ...             3.486096e-40                3.411401e-07
enough and\nlistened to the arguments, I could ...                         0.878680               2.302363e-20  ...             5.866410e-40                3.565728e-07
.  Or, as you brought out,\n> if whatever is ri...                         0.878953               3.004052e-20  ...             5.977738e-40                3.566668e-07
> *******************************                                          0.647793               5.664651e-17  ...             1.805073e-19                4.612731e-04

API Reference

turftopic.late.LateWrapper

Bases: LateModel

Wraps existing Turftopic model so that they can accept and create multi-vector document representations.

Parameters:

Name Type Description Default
model TransformerMixin

Turftopic model to turn into late-interaction model.

required
span_extractor Optional[SpanExtractor]

Object that extracts arbitrary spans from a piece of text. These spans will then be used to pool token embeddings. This could for instance be a SentenceSeparator.

None
include_edge_tokens bool

Indicates whether the tokens on the edges of spans (ones that go over the span boundary) should be included in the span embedding. Ignored when not using a span extractor.

True
window_size Optional[int]

Size of the sliding window to average tokens over. If None, documents will be represented at a token level.

None
step_size Optional[int]

Step size of the window. If (step_size == None) or (step_size == window_size), then windows are separate. If step_size < window_size, windows will overlap. If step_size > window_size, there will be gaps between the windows. In this case, we throw a warning, as this is probably unintended behaviour.

None
pooling Optional[Callable]

Indicates whether and how to pool document-topic matrices. If None, multi-vector topic proportions are returned in a ragged array. If Callable, multiple vectors are averaged with the callable in each document. You could for example take the mean by specifying pooling=np.nanmean.

None
Source code in turftopic/late.py
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
class LateWrapper(LateModel):
    """Wraps existing Turftopic model so that they can accept and create
    multi-vector document representations.

    Parameters
    ----------
    model
        Turftopic model to turn into late-interaction model.
    span_extractor: SpanExtractor or "sentence", default None
        Object that extracts arbitrary spans from a piece of text.
        These spans will then be used to pool token embeddings.
        This could for instance be a SentenceSeparator.
    include_edge_tokens: bool, default True
        Indicates whether the tokens on the edges of spans (ones that go over the span boundary)
        should be included in the span embedding.
        Ignored when not using a span extractor.
    window_size: int, default None
        Size of the sliding window to average tokens over.
        If None, documents will be represented at a token level.
    step_size: int, default None
        Step size of the window.
        If (step_size == None) or (step_size == window_size), then windows are separate.
        If step_size < window_size, windows will overlap.
        If step_size > window_size, there will be gaps between the windows.
        In this case, we throw a warning, as this is probably unintended behaviour.
    pooling: Callable, default None
        Indicates whether and how to pool document-topic matrices.
        If None, multi-vector topic proportions are returned in a ragged array.
        If Callable, multiple vectors are averaged with the callable in each document.
        You could for example take the mean by specifying `pooling=np.nanmean`.
    """

    def __init__(
        self,
        model: TransformerMixin,
        span_extractor: Optional[SpanExtractor] = None,
        include_edge_tokens: bool = True,
        window_size: Optional[int] = None,
        step_size: Optional[int] = None,
        pooling: Optional[Callable] = None,
    ):
        self.model = model
        self.pooling = pooling
        self.window_size = window_size
        self.step_size = step_size
        self.span_extractor = span_extractor
        self.include_edge_tokens = include_edge_tokens
        self.encode_kwargs = self.model.encode_kwargs
        self.trf_kwargs = self.model.trf_kwargs
        self.encode_kwargs = self.model.encode_kwargs
        # Reloading encoder to make sure it is late interaction
        self.encoder_ = self.model.encoder_
        if isinstance(self.encoder_, SentenceTransformer):
            # This is a nasty trick, and I should probably find something better in the future,
            # but this casts a sentence transformer directly into a LateSentenceTransformer
            self.encoder_.__class__ = LateSentenceTransformer

    def transform(
        self,
        raw_documents: list[str],
        embeddings: list[np.ndarray] = None,
        offsets: list[Offsets] = None,
    ):
        if (embeddings is None) or (offsets is None):
            embeddings, offsets = self.encode_late(raw_documents)
        flat_embeddings, lengths = self.flatten_repr(embeddings)
        chunks = get_document_chunks(raw_documents, offsets)
        out_array = self.model.transform(chunks, embeddings=flat_embeddings)
        if self.pooling is None:
            return unflatten_repr(out_array, lengths), offsets
        else:
            return pool_flat(out_array, lengths)

    def fit_transform(
        self,
        raw_documents: list[str],
        y=None,
        embeddings: list[np.ndarray] = None,
        offsets: list[Offsets] = None,
    ):
        if (embeddings is None) or (offsets is None):
            embeddings, offsets = self.encode_late(raw_documents)
        flat_embeddings, lengths = self.flatten_repr(embeddings)
        chunks = get_document_chunks(raw_documents, offsets)
        out_array = self.model.fit_transform(
            chunks, embeddings=flat_embeddings
        )
        if self.pooling is None:
            return unflatten_repr(out_array, lengths), offsets
        else:
            return pool_flat(out_array, lengths)

    @property
    def components_(self):
        return self.model.components_

    @property
    def hierarchy(self):
        return self.model.hierarchy

    @property
    def topic_names(self):
        return self.model.topic_names

    @property
    def classes_(self):
        return self.model.classes_

    @property
    def vectorizer(self):
        return self.model.vectorizer