Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
Quality evaluation strategies for synthetic code-switched data in support of language learning applications
Thesis (PhD (Computer Science))--University of Pretoria, 2025.
Saved in:
| Other Authors: | |
|---|---|
| Format: | Thesis |
| Language: | English |
| Published: |
University of Pretoria
2026
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| _version_ | 1869483796667039744 |
|---|---|
| access_status_str | Open Access |
| author2 | Marivate, Vukosi |
| author_browse | Marivate, Vukosi |
| author_facet | Marivate, Vukosi |
| collection | Thesis |
| dc_rights_str_mv | © 2024 University of Pretoria. All rights reserved. The copyright in this work vests in the University of Pretoria. No part of this work may be reproduced or transmitted in any form or by any means, without the prior written permission of the University of Pretoria. |
| description | Thesis (PhD (Computer Science))--University of Pretoria, 2025. |
| format | Thesis |
| id | oai:repository.up.ac.za:2263/107731 |
| institution | University of Pretoria (South Africa) |
| language | English |
| last_indexed | 2026-07-01T04:04:41.819Z |
| license_str | Other — see source repository |
| provenance_str_mv | Harvested via OAI-PMH from UPSpace — University of Pretoria Institutional Repository |
| publishDate | 2026 |
| publishDateRange | 2026 |
| publishDateSort | 2026 |
| publisher | University of Pretoria |
| publisherStr | University of Pretoria |
| record_format | dspace |
| source_str | UPSpace — University of Pretoria Institutional Repository |
| spelling | oai:repository.up.ac.za:2263/107731 Quality evaluation strategies for synthetic code-switched data in support of language learning applications Marivate, Vukosi michelle.terblanche@gmail.com Olaleye, Kayode Kotzé, Michelle UCTD Sustainable Development Goals (SDGs) Afrikaans-English code-switching Educational NLP Syntactic quality evaluation Large language models Explainability Thesis (PhD (Computer Science))--University of Pretoria, 2025. In a world where almost half of the population is bilingual and a significant proportion is multilingual, code-switching is a common linguistic phenomenon in both spoken and written communication. Increasingly recognised as an intentional and structured practice rather than a linguistic deficiency, code-switching research remains constrained by the lack of high-quality, annotated resources and appropriate evaluation methodologies. These limitations are particularly pronounced in educational natural language processing technologies, where code-switching plays an important role in language learning applications. This thesis addresses the issue of code-switched data scarcity and the challenge of evaluating its syntactic quality. Large language models (LLMs) were leveraged to generate synthetic Afrikaans--English code-switched data using carefully designed prompt templates. The generated data was assessed through human evaluation and LLM-as-a-judge. Findings show that GPT-4o produced more acceptable sentences compared to Gemini 2.0 Flash, while LLM-as-a-judge revealed notable inconsistencies regarding the ability of LLMs to serve as fair evaluators of quality. Word-level language identification and part-of-speech tagging were further introduced, with rule-based and joint tagger approaches outperforming LLM-based methods. To support evaluation of syntactic quality, two evaluation methodologies were explored: a heuristic model and BERT-based classifiers. The heuristic model, based on explicit grammar rules for Afrikaans--English code-switching, achieved strong baseline performance (90.5% F1 score). A multi-input XLM-R classifier using word embeddings, language identification and part-of-speech tags, achieved the highest performance with a 97.5% F1 score. Due to the black-box nature of XLM-R models, interpretability was further explored through gradient- and occlusion-based attribution analyses, which in some cases provided complementary insights into classifier behaviour. However, it remains unclear whether the model consistently relies on linguistically appropriate signals. The final proposed evaluation strategy is a hybrid framework that combines the heuristic rule-based model with the multi-input XLM-R classifier into a meta-classifier, and optimised to reduce false negatives (incorrect CS sentences misclassified as correct). This hybrid model achieved a 93.2% F1 score with 95.0% recall on the incorrect class, balancing rule-based precision with neural generalisation. The main contributions of this thesis include (i) a framework for synthetic code-switched text generation with LLMs, (ii) a curated Afrikaans--English code-switched data set, (iii) a comprehensive evaluation strategy integrating grammar rules, neural models and explainability techniques, and (iv) a proposed set of requirements for a data set tailored to first/second language learning applications. Collectively, these contributions advance both the generation and evaluation of code-switched data, while laying a foundation for future research into educational technologies for code-switching. Computer Science PhD (Computer Science) Unrestricted Faculty of Engineering, Built Environment and Information Technology SDG-04: Quality education 2026-01-30T08:14:18Z 2026-01-30T08:14:18Z 2026 2025 Thesis * A2026 http://hdl.handle.net/2263/107731 https://doi.org/10.25403/UPresearchdata.31077619 en © 2024 University of Pretoria. All rights reserved. The copyright in this work vests in the University of Pretoria. No part of this work may be reproduced or transmitted in any form or by any means, without the prior written permission of the University of Pretoria. application/pdf University of Pretoria |
| spellingShingle | UCTD Sustainable Development Goals (SDGs) Afrikaans-English code-switching Educational NLP Syntactic quality evaluation Large language models Explainability Quality evaluation strategies for synthetic code-switched data in support of language learning applications |
| title | Quality evaluation strategies for synthetic code-switched data in support of language learning applications |
| title_full | Quality evaluation strategies for synthetic code-switched data in support of language learning applications |
| title_fullStr | Quality evaluation strategies for synthetic code-switched data in support of language learning applications |
| title_full_unstemmed | Quality evaluation strategies for synthetic code-switched data in support of language learning applications |
| title_short | Quality evaluation strategies for synthetic code-switched data in support of language learning applications |
| title_sort | quality evaluation strategies for synthetic code switched data in support of language learning applications |
| topic | UCTD Sustainable Development Goals (SDGs) Afrikaans-English code-switching Educational NLP Syntactic quality evaluation Large language models Explainability |
| url | http://hdl.handle.net/2263/107731 https://doi.org/10.25403/UPresearchdata.31077619 |