Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
Conditional random fields for noisy text normalisation
Thesis (MScEng) -- Stellenbosch University, 2014.
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | en_ZA |
| Published: |
Stellenbosch : Stellenbosch University
2015
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| _version_ | 1867613923135455232 |
|---|---|
| access_status_str | Open Access |
| author | Coetsee, Dirko |
| author2 | Du Preez, J. A. |
| author_browse | Coetsee, Dirko Du Preez, J. A. |
| author_facet | Du Preez, J. A. Coetsee, Dirko |
| author_sort | Coetsee, Dirko |
| collection | Thesis |
| dc_rights_str_mv | Stellenbosch University |
| description | Thesis (MScEng) -- Stellenbosch University, 2014. |
| format | Thesis |
| id | oai:scholar.sun.ac.za:10019.1/96064 |
| institution | Stellenbosch University (South Africa) |
| language | en_ZA |
| last_indexed | 2026-06-10T12:43:50.825Z |
| license_str | Other — see source repository |
| provenance_str_mv | Harvested via OAI-PMH from SUNScholar — Stellenbosch University Repository |
| publishDate | 2015 |
| publishDateRange | 2015 |
| publishDateSort | 2015 |
| publisher | Stellenbosch : Stellenbosch University |
| publisherStr | Stellenbosch : Stellenbosch University |
| record_format | dspace |
| source_str | SUNScholar — Stellenbosch University Repository |
| spelling | oai:scholar.sun.ac.za:10019.1/96064 Conditional random fields for noisy text normalisation Coetsee, Dirko Du Preez, J. A. Stellenbosch University. Faculty of Engineering. Department of Electrical and Electronic Engineering. Conditional random fields Noisy text Spelling correction Mikroblogging Unstructured data Data mining Text normalisation Tokeniser UCTD Thesis (MScEng) -- Stellenbosch University, 2014. ENGLISH ABSTRACT: The increasing popularity of microblogging services such as Twitter means that more and more unstructured data is available for analysis. The informal language usage in these media presents a problem for traditional text mining and natural language processing tools. We develop a pre-processor to normalise this noisy text so that useful information can be extracted with standard tools. A system consisting of a tokeniser, out-of-vocabulary token identifier, correct candidate generator, and N-gram language model is proposed. We compare the performance of generative and discriminative probabilistic models for these different modules. The effect of normalising the training and testing data on the performance of a tweet sentiment classifier is investigated. A linear-chain conditional random field, which is a discriminative model, is found to work better than its generative counterpart for the tokenisation module, achieving a 0.76% character error rate compared to 1.41% for the finite state automaton. For the candidate generation module, however, the generative weighted finite state transducer works better, getting the correct clean version of a word right 36% of the time on the first guess, while the discriminatively trained hidden alignment conditional random field only achieves 6%. The use of a normaliser as a pre-processing step does not significantly affect the performance of the sentiment classifier. AFRIKAANSE OPSOMMING: Mikro-webjoernale soos Twitter word al hoe meer gewild, en die hoeveelheid ongestruktureerde data wat beskikbaar is vir analise groei daarom soos nooit tevore nie. Die informele taalgebruik in hierdie media maak dit egter moeilik om tradisionele tegnieke en bestaande dataverwerkingsgereedskap toe te pas. ’n Stelsel wat hierdie ruiserige teks normaliseer word ontwikkel sodat bestaande pakkette gebruik kan word om die teks verder te verwerk. Die stelsel bestaan uit ’n module wat die teks in woordeenhede opdeel, ’n module wat woorde identifiseer wat gekorrigeer moet word, ’n module wat dan kandidaat korreksies voorstel, en ’n module wat ’n taalmodel toepas om die mees waarskynlike skoon teks te vind. Die verrigting van diskriminatiewe en generatiewe modelle vir ’n paar van hierdie modules word vergelyk en die invloed wat so ’n normaliseerder op die akkuraatheid van ’n sentimentklassifiseerder het word ondersoek. Ons bevind dat ’n lineêre-ketting voorwaardelike toevalsveld—’n diskriminatiewe model — beter werk as sy generatiewe eweknie vir tekssegmentering. Die voorwaardelike toevalsveld-model behaal ’n karakterfoutkoers van 0.76%, terwyl die toestandsmasjien-model 1.41% behaal. Die toestantsmasjien-model werk weer beter om kandidaat woorde te genereer as die verskuilde belyningsmodel wat ons geïmplementeer het. Die toestandsmasjien kry 36% van die tyd die regte weergawe van ’n woord met die eerste raaiskoot, terwyl die diskriminatiewe model dit slegs 6% van die tyd kan doen. Laastens het ons bevind dat die vooraf normalisering van Twitter boodskappe nie ’n beduidende effek op die akkuraatheid van ’n sentiment klassifiseerder het nie. 2015-01-13T11:50:02Z 2015-01-13T11:50:02Z 2014-12 Thesis http://hdl.handle.net/10019.1/96064 en_ZA Stellenbosch University xv, 145 p. : ill. application/pdf Stellenbosch : Stellenbosch University |
| spellingShingle | Conditional random fields Noisy text Spelling correction Mikroblogging Unstructured data Data mining Text normalisation Tokeniser UCTD Coetsee, Dirko Conditional random fields for noisy text normalisation |
| title | Conditional random fields for noisy text normalisation |
| title_full | Conditional random fields for noisy text normalisation |
| title_fullStr | Conditional random fields for noisy text normalisation |
| title_full_unstemmed | Conditional random fields for noisy text normalisation |
| title_short | Conditional random fields for noisy text normalisation |
| title_sort | conditional random fields for noisy text normalisation |
| topic | Conditional random fields Noisy text Spelling correction Mikroblogging Unstructured data Data mining Text normalisation Tokeniser UCTD |
| url | http://hdl.handle.net/10019.1/96064 |
| work_keys_str_mv | AT coetseedirko conditionalrandomfieldsfornoisytextnormalisation |