Full Text Available
Note: Clicking the button above will open the full text document at the original institutional repository in a new window.
A vision-based South African sign language tutor
Thesis (PhD)--Stellenbosch University, 2014.
Saved in:
| Main Author: | |
|---|---|
| Other Authors: | |
| Format: | Thesis |
| Language: | en_ZA |
| Published: |
Stellenbosch : Stellenbosch University
2014
|
| Subjects: | |
| Tags: |
No Tags, Be the first to tag this record!
|
| _version_ | 1867613917889429504 |
|---|---|
| access_status_str | Open Access |
| author | De Villiers, Hendrik Adrianus Cornelis |
| author2 | Niesler, T. R. |
| author_browse | De Villiers, Hendrik Adrianus Cornelis Niesler, T. R. |
| author_facet | Niesler, T. R. De Villiers, Hendrik Adrianus Cornelis |
| author_sort | De Villiers, Hendrik Adrianus Cornelis |
| collection | Thesis |
| dc_rights_str_mv | Stellenbosch University |
| description | Thesis (PhD)--Stellenbosch University, 2014. |
| format | Thesis |
| id | oai:scholar.sun.ac.za:10019.1/86333 |
| institution | Stellenbosch University (South Africa) |
| language | en_ZA |
| last_indexed | 2026-06-10T12:43:46.104Z |
| license_str | Other — see source repository |
| provenance_str_mv | Harvested via OAI-PMH from SUNScholar — Stellenbosch University Repository |
| publishDate | 2014 |
| publishDateRange | 2014 |
| publishDateSort | 2014 |
| publisher | Stellenbosch : Stellenbosch University |
| publisherStr | Stellenbosch : Stellenbosch University |
| record_format | dspace |
| source_str | SUNScholar — Stellenbosch University Repository |
| spelling | oai:scholar.sun.ac.za:10019.1/86333 A vision-based South African sign language tutor De Villiers, Hendrik Adrianus Cornelis Niesler, T. R. Van Zijl, L. Stellenbosch University. Faculty of Engineering. Dept. of Electrical and Electronic Engineering. Computer vision Sign language -- Study and teaching Sign language -- Machine translating Pattern recognition systems Theses -- Electrical and electronic engineering Dissertations -- Electrical and electronic engineering UCTD Thesis (PhD)--Stellenbosch University, 2014. ENGLISH ABSTRACT: A sign language tutoring system capable of generating detailed context-sensitive feedback to the user is presented in this dissertation. This stands in contrast with existing sign language tutor systems, which lack the capability of providing such feedback. A domain specific language is used to describe the constraints placed on the user’s movements during the course of a sign, allowing complex constraints to be built through the combination of simpler constraints. This same linguistic description is then used to evaluate the user’s movements, and to generate corrective natural language feedback. The feedback is dynamically tailored to the user’s attempt, and automatically targets that correction which would require the least effort on the part of the user. Furthermore, a procedure is introduced which allows feedback to take the form of a simple to-do list, despite the potential complexity of the logical constraints describing the sign. The system is demonstrated using real video sequences of South African Sign Language signs, exploring the different kinds of advice the system can produce, as well as the accuracy of the comments produced. To provide input for the tutor system, the user wears a pair of coloured gloves, and a video of their attempt is recorded. A vision-based hand pose estimation system is proposed which uses the Earth Mover’s Distance to obtain hand pose estimates from images of the user’s hands. A two-tier search strategy is employed, first obtaining nearest neighbours using a simple, but related, metric. It is demonstrated that the two-tier system’s accuracy approaches that of a global search using only the Earth Mover’s Distance, yet requires only a fraction of the time. The system is shown to outperform a closely related system on a set of 500 real images of gloved hands. AFRIKAANSE OPSOMMING: ’n Gebaretaaltutorstelsel met die vermo¨e om konteks-sensitiewe terugvoer te lewer aan die gebruiker word uiteengesit in hierdie proefskrif. Hierdie staan in kontras met bestaande tutorstelsels, wat nie hierdie kan bied vir die gebruiker nie. ’n Domein-spesifieke taal word gebruik om beperkinge te definieer op die gebruiker se bewegings deur die loop van ’n gebaar. Komplekse beperkinge kan opgebou word uit eenvoudiger beperkinge. Dieselfde linguistieke beskrywing van die gebaar word gebruik om die gebruiker se bewegings te evalueer, en om korrektiewe terugvoer te genereer in teksvorm. Die terugvoer word dinamies aangepas met betrekking tot die gebruiker se probeerslag, en bepaal outomaties die maklikste manier wat die gebruiker sy/haar fout kan korrigeer. ’n Prosedure word uiteengesit om die terugvoer in ’n eenvoudige lysvorm aan te bied, ongeag die kompleksiteit van die linguistieke beskrywing van die gebaar. Die stelsel word gedemonstreer aan die hand van opnames van gebare uit Suid-Afrikaanse Gebaretaal. Die verskeie tipes terugvoer wat die stelsel kan lewer, asook die akkuraatheid van hierdie terugvoer, word ondersoek. Om vir die tutorstelsel intree te bied, dra die gebruiker ’n stel gekleurde handskoene. ’n Visie-gebaseerde handvormafskattingstelsel wat gebruik maak van die Aardverskuiwersafstand (Earth Mover’s Distance) word voorgestel. ’n Twee-vlak soekstrategie word gebruik. ’n Rowwe afstandsmate word gebruik om ’n stel voorlopige handpostuurkandidate te verkry, waarna die stel verfyn word deur gebruik van die Aardverskuiwersafstand. Dit word gewys dat hierdie benaderde strategie se akkuraatheid grens aan die van eksakte soektogte, maar neem slegs ’n fraksie van die tyd. Toetsing op ’n stel van 500 re¨ele beelde, wys dat hierdie stelsel beter presteer as ’n naverwante stelsel uit die literatuur. Doctoral 2014-04-16T17:28:51Z 2014-04-16T17:28:51Z 2014-04 Thesis http://hdl.handle.net/10019.1/86333 en_ZA Stellenbosch University xiii, 158 p. : ill. application/pdf Stellenbosch : Stellenbosch University |
| spellingShingle | Computer vision Sign language -- Study and teaching Sign language -- Machine translating Pattern recognition systems Theses -- Electrical and electronic engineering Dissertations -- Electrical and electronic engineering UCTD De Villiers, Hendrik Adrianus Cornelis A vision-based South African sign language tutor |
| title | A vision-based South African sign language tutor |
| title_full | A vision-based South African sign language tutor |
| title_fullStr | A vision-based South African sign language tutor |
| title_full_unstemmed | A vision-based South African sign language tutor |
| title_short | A vision-based South African sign language tutor |
| title_sort | vision based south african sign language tutor |
| topic | Computer vision Sign language -- Study and teaching Sign language -- Machine translating Pattern recognition systems Theses -- Electrical and electronic engineering Dissertations -- Electrical and electronic engineering UCTD |
| url | http://hdl.handle.net/10019.1/86333 |
| work_keys_str_mv | AT devilliershendrikadrianuscornelis avisionbasedsouthafricansignlanguagetutor AT devilliershendrikadrianuscornelis visionbasedsouthafricansignlanguagetutor |