[NLP] Pre-Processing Tasks - ver.dBRang

/*******************************************************************************************************************
-- Title : [NLP] Pre-Processing Tasks - ver.dBRang
-- Key word : pre-processing preprocessing nlp pipeline 자연어처리 파이프라인 전처리 
*******************************************************************************************************************/

■ NLP Pipeline
    ☞ dbrang.tistory.com/1259   

■ Pre-processing Tasks

■ Word Unit

 

Parsing & Encoding
Remove Tags	정규 표현식으로 태그 제거(1149) 정규 표현식(정규식) 설명(1152)
Deal with Encoding	한글 깨짐 이슈(934)
Tokenization
Sentence Tokenizing	Nltk(1183, 1131, 1193) RE(1183, 1184) SpaCy(1244, 1246) TextBlob(1199) Custom(1140)
Word Tokenizing	Nltk(1183, 1131, 1193, 1255) RE(1183, 1184, 1205) SpaCy(1248, 1244, 1249) Scikit-Learn(1231, 1197, 1253) TextBlob(1199, 1198) Custom(1248, 1265)
Stop word
Stopwords	Nltk(1132, 1184, 1205) Stop_Words(1255) Scikit-Learn(1263) Custom(1263, 1265)
POS(Part-Of-Speech) Tagging
Tag List	Nltk(1139) Universal(1245)
POS Tagging	Nltk(1140, 1189, 1193, 1195, 1205, 1173) Stanford(1173) SpaCy(1244) TextBlob(1198, 1199)
한글 Tagging	KLT(1273) Twitter(1274)
Lemmatization
Stemming	Nltk(1138, 1154, 1255) Stemming vs. Lemmatizing(1171)
Lemmatizing	Nltk(1154, 1189, 1205) SpaCy(1244, 1248, 1249) TextBlob(1199)
Named Entity Recognition
NE Recognition	Nltk(1148) SpaCy(1244, 1246)

 

 

 

■ Phrase Unit

Chunking
Chunk	Nltk(1145, 1193, 1194, 1195) SpaCy(1244, 1246, 1248) TextBlob(1198, 1199, 1200)
Cohesion
Cohesion	Cohesion Theory Cohesion