Preserve paragraph divisions of documents

[wuqui]

Hello,

I have documents that are structured as paragraphs as they are cleaned versions of scraped html pages. What's the best / most spaCy way of preserving the paragraph structure (while still having one Doc per file), so that I can later refer back to paragraph units for exporting xml or doing text classification on paragraph level? Should I create 'paragraph Spans' or something like that?

Thanks!
Quirin

[ines]

Should I create 'paragraph Spans' or something like that?

Yes, that sounds like a good solution. If you have the character offsets of the paragraphs, you can use Doc.char_span to create the spans.

If you want to do this more elegantly, you could also store the paragraph offsets in the doc.user_data and then add a custom Doc attribute like doc._.paragraphs that returns (or yields) the paragraph spans for a given Doc.

For example:

doc = nlp(paragraph_text)
doc.user_data["paragraph_offsets"] = [(0, 392), (393, 848)] # etc.

from spacy.tokens import Doc

def get_paragraphs(doc):
    offsets = doc.user_data.get("paragraph_offsets", [])
    return [doc.char_span(start, end) for start, end in offsets]

Doc.set_extension("paragraphs", getter=get_paragraphs)

[wuqui]

Thanks a lot for your quick and helpful reply! It works :)

Is it possible to access the paragraphs' sentences? .sents doesn't work, as Spans are probably meant to be used for sub-sentential patterns.

[ines]

Is it possible to access the paragraphs' sentences? .sents doesn't work, as Spans are probably meant to be used for sub-sentential patterns.

Yeah, there's Span.sent, but no Span.sents. However, the Doc.sents are also just spans, and they all expose a start_char/end_char (character offsets) and start/end (token offsets). So you could get those and for each paragraph span, check whether the sentence start and end is within the paragraph start and end. If so, you know that the paragraph contains the sentence.