Towards Zero-shot Sign Languge Recognition

Yunus Can Bilge R. Gokberk Cinbis Nazli Ikizler-Cinbis

The paper tackles the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign classes to recognize the instances of unseen sign classes. In this context, readily available textual sign descriptions and attributes collected from sign language dictionaries are utilized as semantic class representations for knowledge transfer. For this novel problem setup,

(i) We introduce three benchmark datasets with their accompanying textual and attribute descriptions to analyze the problem in detail.
(ii) Our proposed approach builds spatio-temporal models of body and hand regions.
(ii) By leveraging the descriptive text and attribute embeddings along with these visual representations within a zero-shot learning framework, we show that textual and attribute based class definitions can provide effective knowledge for the recognition of previously unseen sign classes.
(iii) By leveraging the descriptive text and attribute embeddings along with these visual representations within a zero-shot learning framework, we show that textual and attribute based class definitions can provide effective knowledge for the recognition of previously unseen sign classes.
(iv) We additionally introduce techniques to analyze the influence of binary attributes in correct and incorrect zero-shot predictions.

We anticipate that the introduced approaches and the accompanying datasets will provide a basis for further exploration of zero-shot learning in sign language recognition.

Dataset Samples for the ASL-Text and MS-ZSSLR-W/C

BICYLE EAT ALONE LIBRARY

Move both S hands in alternating forward circles, palms facing down, in front of each side of the body.

Bring the fingertips of the right flattened O hand, palm facing in, to the lips with a repeated movement.

With the right index finger extended up, move the right hand, palm facing back, in a small repeated circle in front of the right shoulder.

Move the right L hand, palm facing forward, in a circle in front of the right shoulder.

BOOK HORSE ORANGE FISH

Beginning with the palms of both open hands together in front of the chest, fingers angled forward, bring the hands apart at the top while keeping the little fingers together.

With the extended thumb of the right U hand against the right side of the forehead, palm facing forward, bend the fingers of the U hand up and down with a double movement.

Beginning with the right C hand in front of the mouth, palm facing left, squeeze the fingers open and closed with a repeated movement, forming an S hand each time.

While touching the wrist of the right open hand, palm facing left, with the extended left index finger, swing the right hand back and forth with a double movement.

Citation

Please cite if you make use of this work and/or the dataset:

        
          @ARTICLE {9681230,
                author = {Y. Bilge and R. Cinbis and N. Ikizler-Cinbis},
                journal = {IEEE Transactions on Pattern Analysis & Machine Intelligence},
                title = {Towards Zero-Shot Sign Language Recognition},
                year = {2023},
                volume = {45},
                number = {01},
                issn = {1939-3539},
                pages = {1217-1232},
                doi = {10.1109/TPAMI.2022.3143074},
                publisher = {IEEE Computer Society},
                address = {Los Alamitos, CA, USA},
                month = {jan}
              }

Based on the template by Steve Smith.