In order to get the most out of the package, let’s enumerate a few things one can now easily do with your text annotated using the udpipe package using merely the Parts of Speech tags & the Lemma of each word.
In particular, in this report we focus on basic analytical use cases of pos tagging, lemmatisation and co-occurrences where we will show in this vignette some basic frequency statistics which can be extracted without any hassle once you have annotated your text. Furthermore, the logic accounts for all languages and is language-agnostic.
Let’s start by annotating some text in Spanish. The annotated data.frame can next be used for basic text analytics.
The resulting data.frame has a field called
upos which is the Universal Parts of Speech tag and also a field called
lemma which is the root form of each token in the text. These 2 fields give us a broad range of analytical possibilities.
In most languages, nouns (NOUN) are the most common types of words, next to verbs (VERB) and these are the most relevant for analytical purposes, next to the adjectives (ADJ) and proper nouns (PROPN). For a detailed list of all POS tags: visit https://universaldependencies.org/u/pos/index.html.
Parts of Speech tags are really interesting to extract easily the words you like to plot. You really don’t need stopwords for doing this, just select nouns / verbs or adjectives and you have already the most relevant parts for basic frequency analysis.
Frequency statistics of words are nice but most of the time, you are getting stuck in words which only make sense in combination with other words. Hence you want to find keywords which are a combination of words.
Currently, this R package provides 3 methods to identify keywords in text
## Using RAKE stats <- keywords_rake(x = x, term = "lemma", group = "doc_id", relevant = x$upos %in% c("NOUN", "ADJ")) stats$key <- factor(stats$keyword, levels = rev(stats$keyword)) barchart(key ~ rake, data = head(subset(stats, freq > 3), 20), col = "cadetblue", main = "Keywords identified by RAKE", xlab = "Rake")
## Using Pointwise Mutual Information Collocations x$word <- tolower(x$token) stats <- keywords_collocation(x = x, term = "word", group = "doc_id") stats$key <- factor(stats$keyword, levels = rev(stats$keyword)) barchart(key ~ pmi, data = head(subset(stats, freq > 3), 20), col = "cadetblue", main = "Keywords identified by PMI Collocation", xlab = "PMI (Pointwise Mutual Information)")
## Using a sequence of POS tags (noun phrases / verb phrases) x$phrase_tag <- as_phrasemachine(x$upos, type = "upos") stats <- keywords_phrases(x = x$phrase_tag, term = tolower(x$token), pattern = "(A|N)*N(P+D*(A|N)*N)*", is_regex = TRUE, detailed = FALSE) stats <- subset(stats, ngram > 1 & freq > 3) stats$key <- factor(stats$keyword, levels = rev(stats$keyword)) barchart(key ~ freq, data = head(stats, 20), col = "cadetblue", main = "Keywords - simple noun phrases", xlab = "Frequency")
Co-occurrences allow to see how words are used either in the same sentence or next to each other. This R package make creating co-occurrence graphs using the relevant Parts of Speech tags as easy as possible.
In this example we look how many times nouns and adjectives are used in the same sentence.
term1 term2 cooc 1 barrio tranquilo 48 2 centro minuto 44 3 apartamento centro 38 4 apartamento limpio 36 5 estacion minuto 34 6 estacion tren 34
The result can be easily visualised using the ggraph R package which can visualise the word network.
library(igraph) library(ggraph) library(ggplot2) wordnetwork <- head(cooc, 30) wordnetwork <- graph_from_data_frame(wordnetwork) ggraph(wordnetwork, layout = "fr") + geom_edge_link(aes(width = cooc, edge_alpha = cooc), edge_colour = "pink") + geom_node_text(aes(label = name), col = "darkgreen", size = 4) + theme_graph(base_family = "Arial Narrow") + theme(legend.position = "none") + labs(title = "Cooccurrences within sentence", subtitle = "Nouns & Adjective")
If you are interested in visualising which words follow one another. This can be done by calculating word cooccurrences of a specific Parts of Speech type which follow one another where you can specify how far away you want to look regarding ‘following one another’ (in the example below we indicate skipgram = 1 which means look to the next word and the word after that).
term1 term2 cooc 1 barrio tranquilo 27 2 estacion tren 27 3 transporte publico 23 4 pleno centro 20 5 estacion central 16 6 parada metro 15
Once you have these coocccurrences, you can easily perform the same plotting as above.
library(igraph) library(ggraph) library(ggplot2) wordnetwork <- head(cooc, 15) wordnetwork <- graph_from_data_frame(wordnetwork) ggraph(wordnetwork, layout = "fr") + geom_edge_link(aes(width = cooc, edge_alpha = cooc)) + geom_node_text(aes(label = name), col = "darkgreen", size = 4) + theme_graph(base_family = "Arial Narrow") + labs(title = "Words following one another", subtitle = "Nouns & Adjective")
For details on the visualisation of networks, visit the igraph and ggraph packages. The network plots shown above are just run once to show basic functionality, they are not run in the vignette in order to reduce package dependencies and to leave the freedom to people to choose their preferred visualisation toolkit.
Keyword correlations indicate how terms are just together in a same document/sentence. While co-occurrences focus on frequency, correlation measures between 2 terms can also be high even if 2 terms occur only a small number of times but always appear together.
In the below example, we show how nouns and adjectives are correlated within each sentence of a document.
x$id <- unique_identifier(x, fields = c("sentence_id", "doc_id")) dtm <- subset(x, upos %in% c("NOUN", "ADJ")) dtm <- document_term_frequencies(dtm, document = "id", term = "lemma") dtm <- document_term_matrix(dtm) dtm <- dtm_remove_lowfreq(dtm, minfreq = 5) termcorrelations <- dtm_cor(dtm) y <- as_cooccurrence(termcorrelations) y <- subset(y, term1 < term2 & abs(cooc) > 0.2) y <- y[order(abs(y$cooc), decreasing = TRUE), ] head(y)
term1 term2 cooc 78018 grand place 0.7738751 100575 publico transporte 0.7485261 99273 sabano toalla 0.6281728 91868 fin semana 0.5534800 33077 cuenta equipaje 0.5465754 89264 instrucción ropa 0.5336612
Need support in text mining. Contact BNOSAC: http://www.bnosac.be