"Retaining selected word pairs when tokenizing"
MavenWhen tokenizing into single word tokens, is there a way to keep selected pairs of words together as a single token?
For example, in soccer the term "centre forward" makes more sense as a single token. I looked at n-grams, but this pairs words that I do not want to pair. I tried using the stem dictionary, but this seems not to work across multiple tokens, and if I put the stem before tokenize, e.g. to change centre forward to centre-forward, this doesn't appear to work.
Best Answer
-
IngoRM
Employee-RapidMiner, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 
RM Founder
Hi Carl,
All observations are correct. Since there is no replace operator across multiple tokens, I think you have to apply the Replace operator on the data set in your case. The other options do not seem to be really feasible here.
But don't worry, you can actually do this by first transforming your document into an example set, perfom the replacement, and transform it back into a document. The process below shows you how you can do this. Please note that you either need to change your tokenization to something else than "non letters" or you need to use letters as the delimiter in your replacement (or just no delimiter at all).
This is probably not winning a first price for elegance but it does the job :smileywink:
Hope that helps,
Ingo
<?xml version="1.0" encoding="UTF-8"?><process version="7.3.001">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="7.3.001" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="open_file" compatibility="7.3.001" expanded="true" height="68" name="Open File" width="90" x="45" y="34">
<parameter key="filename" value="C:\Users\IngoMierswa\Desktop\Latest Materials\Data\mini_newsgroups\mini_newsgroups\alt.atheism\51121"/>
</operator>
<operator activated="true" class="text:read_document" compatibility="7.3.000" expanded="true" height="68" name="Read Document" width="90" x="179" y="34"/>
<operator activated="true" class="text:documents_to_data" compatibility="7.3.000" expanded="true" height="82" name="Documents to Data" width="90" x="313" y="34">
<parameter key="text_attribute" value="text"/>
</operator>
<operator activated="true" class="replace" compatibility="7.3.001" expanded="true" height="82" name="Replace" width="90" x="447" y="34">
<parameter key="attribute_filter_type" value="single"/>
<parameter key="attribute" value="text"/>
<parameter key="replace_what" value="political correctness"/>
<parameter key="replace_by" value="politicalDELIMcorrectness"/>
</operator>
<operator activated="true" class="text:data_to_documents" compatibility="7.3.000" expanded="true" height="68" name="Data to Documents" width="90" x="581" y="34">
<list key="specify_weights"/>
</operator>
<operator activated="true" class="select" compatibility="7.3.001" expanded="true" height="68" name="Select" width="90" x="715" y="34"/>
<operator activated="true" class="text:tokenize" compatibility="7.3.000" expanded="true" height="68" name="Tokenize" width="90" x="849" y="34">
<parameter key="characters" value=".: "/>
</operator>
<connect from_op="Open File" from_port="file" to_op="Read Document" to_port="file"/>
<connect from_op="Read Document" from_port="output" to_op="Documents to Data" to_port="documents 1"/>
<connect from_op="Documents to Data" from_port="example set" to_op="Replace" to_port="example set input"/>
<connect from_op="Replace" from_port="example set output" to_op="Data to Documents" to_port="example set"/>
<connect from_op="Data to Documents" from_port="documents" to_op="Select" to_port="collection"/>
<connect from_op="Select" from_port="selected" to_op="Tokenize" to_port="document"/>
<connect from_op="Tokenize" from_port="document" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="0"/>
</process>
</operator>
</process>0
Answers
Hi,
you could apply a "Replace" operator before you tokenize. Let's assume your text documents are initially stored as values in a nominal / text column. Then you can use "Replace" to, well, replace "centre forward" by "centre_forward" which will be kept as is in a later tokenization.
Hope that helps,
Ingo
You could do this a couple of different ways. First, you could use n-grams and then a custom stopword dictionary after that to remove the n-grams that you are not interested in (it just requires a text file as input so if you output the word list after the n-gram using "Wordlist to Data" then you should be able to copy/paste the relevant items into a text file fairly easily). This is probably the way I would do it if I had a large number of substitutions to make.
Another approach would be to use a stem dictionary. It sounds like you tried a variation of this, but you would want to place it after the n-gram operator and after tokenize. I don't see why that approach wouldn't work, although I haven't tried it.
A third option if you have only a few of these substitutions to make is simply to use the "replace token" operator, which allows you to use regular expressions for your substitution search.
I hope this helps!
Lindon Ventures
Data Science Consulting from Certified RapidMiner Experts
Hi Brian,
Just to add on the last one: the problem is that "Replace Token" only works on single tokens, so if you already have tokenized the text, the two words are now separated into two tokens and can no longer be replaced...
Cheers,
Ingo
Yep, sorry, I should have clarified that in this instance you can use "replace token" after you have generated n-grams, so you could turn "centre_.*" into "centre_forward" or similar.
Lindon Ventures
Data Science Consulting from Certified RapidMiner Experts
Thanks Brian / Ingo. Sorry, I should have attached my process with the question. Looking at the different options:
1 - Replace before tokenization acts on an example set, and I'm initially processing a document.
2 - n-grams would create too many pairings that I would not be interested in, and if I deleted these, my frequency count would understate certain words if they'd been part of the n-grams.
3 - Stemming after tokenization would require a lot of patterns, and I'd need to re-aggregate the word frequency after breaking up the n-grams that I'm not interested in.
4 - Replace after tokenization and n-graming would have a similar effect as per 3.
For the most part, I'm interested in single words, with just a few exceptions where compound nouns (or concepts) make more sense than the inidividual words. And I wanted to see if I can distill a PDF in as few steps as possible. So ideally a Replace acting early on a document to hyphenate the concepts I want to retain as tokens might be ideal if that were possible.
Thank you. That worked well. I used Replace(Dictionary) so I could create a small number of replacements (via an Excel).