Access SpanCat predictions for a specific span

[baukearends]

After successfully training and testing my model, I would like to access predictions for specific spans. In particular, I'm looking for SpanCat predictions on the exact gold spans that may or may not be present in the docs. I'm able to run a prediction and access the 'raw' probabilities, but for all spans (not just the gold ones) the predicted span is 'no_label'. This does not seem right as I'm receiving predictions for other labels when I call the nlp pipeline, these overlap in general with the gold spans (Jaccard similarity of 0.95).

Can anybody help me understand what's the correct way to access the 'raw' probabilities?

import spacy
from spacy.tokens import DocBin
from spacy.scorer import Scorer
from spacy.training import Example

# Load pre-trained model and data
nlp = spacy.load('/path/to/model')
spancat = nlp.get_pipe("spancat")
doc_bin = DocBin().from_disk('./test.spacy')
docs = list(doc_bin.get_docs(nlp.vocab))
scorer = Scorer(nlp)

# Run spancat model - works fine
examples = []
for doc in docs:
    prediction = nlp(doc.text)
    example = Example(prediction, doc)
    examples.append(example)

# Scores available for all labels
scores = scorer.score(examples)

# Try to access spancat predictions on the exact gold spans
# Prints 'no_label' for all spans
label_rev_map = {v: k for k, v in spancat._label_map.items()}
label_rev_map[spancat._negative_label_i] = 'no_label'
for doc in docs:
    preds = spancat.predict([nlp.tokenizer(doc.text)])
    for span in doc.spans['sc']:
        whole_span_pred = preds[1][[span.start, span.end]]
        predicted_label = label_rev_map[whole_span_pred.argmax()]
        print(predicted_label)

Config:

[paths]
train = "./train.spacy"
dev = "./test.spacy"
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","spancat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat_singlelabel"
allow_overlap = true
negative_weight = 1.0
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_range_suggester.v1"
min_size = 1
max_size = 25

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.spancat]

[initialize.components.spancat.labels]
@readers = "spacy.read_labels.v1"
path = "corpus/labels/spancat.json"
require = false

[initialize.tokenizer]

[svlandeg]

Your pipeline consists of two components, tok2vec and spancat:

pipeline = ["tok2vec","spancat"]

And they both need to be run to create predictions. This is due to the "listener" pattern that is being applied:

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"

In a nutshell, the listener will fetch results that were produced by the tok2vec component earlier on in the pipeline. So that's why

nlp(doc.text)

works - it runs the full pipeline, while

preds = spancat.predict([nlp.tokenizer(doc.text)])

doesn't work correctly - it will not get sensical vectors as input for the spancat component.