Extract paragraph surrounding phrase with spaCy from pandas column

Question

I have a data frame with text data in one column. From this column, I would like to use spaCy to retrieve the sentences that surround a matchword.

Consider this toy data frame:

import pandas as pd
df_test: pd.DataFrame = pd.DataFrame(
    {
        "col1": ["2022-01-01", "2022-10-10", "2022-12-12"],
        "text": [
            "Sentence without the matching word. Another sentence without the matching word.",
            "Sentence with lowercase matchword_one. And a sentence without the matching word. And a sentence with matchword_two.",
            "Sentence with uppercase Matchword_ONE. And another sentence with the uppercase Matchword_one.",
        ],
    }
)

And this phrase matcher containing the two patterns matchw1 and matchw2:

import spacy

nlp = spacy.load("en_core_web_sm")
phrase_matcher = spacy.matcher.PhraseMatcher(nlp.vocab, attr="LOWER")
patterns1 = [nlp(text) for text in ["matchword_one"]]
phrase_matcher.add("matchw1", None, *patterns1)
patterns2 = [nlp(text) for text in ["matchword_two"]]
phrase_matcher.add("matchw2", None, *patterns2)

I now process the text to contain a spacy doc in column text_spacy

df_test['text_spacy'] = [doc for doc in nlp.pipe(df_test['text'].tolist())]  # convert to spacy object
type(df_test.at[0, 'text_spacy']) # check that cell contains a spaCy Doc object

and apply the matcher:

df_test['matches_phrases'] = df_test['text_spacy'].apply(phrase_matcher)  # match patterns

So far, so good. To now retrieve the sentence containing a matchword for a sincgle object, I would use:

doc = nlp(
    "Sentence with lowercase matchword_one. And a sentence without the matching word. And a sentence with matchword_two."
)
for sent in doc.sents:
    for match_id, start, end in phrase_matcher(nlp(sent.text)):
        if nlp.vocab.strings[match_id] in ["matchw1"]:
            print("matchw1", sent.text)
            print("")

        if nlp.vocab.strings[match_id] in ["matchw2"]:
            print("matchw2", sent.text)
            print("")
## Out: matchw1 Sentence with lowercase matchword_one.
## Out: matchw2 And a sentence with matchword_two.

How do I do the same on the column and save the phrase in a column that has the name of the pattern?

The expected output is this:

## expected output:
#                    
#          col1  ...                                    matches_phrases  phrase_matchw1                                                                                phrase_matchw2
# 0  2022-01-01  ...                                                 []                               
# 1  2022-10-10  ...  [(15306160315042522568, 3, 4), (14646110443092...  Sentence with lowercase matchword_one.                                                        And a sentence with matchword_two.
# 2  2022-12-12  ...  [(15306160315042522568, 3, 4), (15306160315042...  Sentence with uppercase Matchword_ONE. And another sentence with the uppercase Matchword_one.       

                

My hunch is it would be something along the lines of df_test['matches_phrases'].apply(lambda x: return x.text if match_id, start, end in x) (which doesn't work but I hope it illustrates the logic.

Many thanks for hints and pointers!

Answer 1

Here is one way to do it:

for pat in ["matchw1", "matchw2"]:
    df_test[f"phrase_{pat}"] = df_test.apply(
        lambda x: " ".join(
            [
                x["text"].split(". ")[i]
                for i, item in enumerate(x["matches_phrases"])
                if nlp.vocab.strings[item[0]] in [pat]
            ]
        ),
        axis=1,
    )

Then:

print(df_test)
# Output
         col1        \
0  2022-01-01   ...   
1  2022-10-10   ...
2  2022-12-12   ...

                                                  matches_phrases  \
0                                                              []   
1  [(15306160315042522568, 3, 4), (14646110443092162848, 17, 18)]   
2  [(15306160315042522568, 3, 4), (15306160315042522568, 11, 12)]   

                                                                                 phrase_matchw1  \
0
1                                                         Sentence with lowercase matchword_one   
2  Sentence with uppercase Matchword_ONE And another sentence with the uppercase Matchword_one.   

                             phrase_matchw2  
0
1  And a sentence without the matching word
2