Malay Language: Linguistic Minimal Pairs
Abstract Malay language model evaluation currently relies on question-and-answer benchmarks and lacks curated minimal pairs for evaluating language model grammaticality. To address this, we introduce the first Malay-specific minimal-pair dataset for language model evaluation, focusing on two phenomena: Verb Affixation (distinguishing passive prefixes di- and diper-) and Reduplication (ensuring head-noun only pluralisation). Our result reveals that our model achieved high perplexity scores but lower SLOR accuracy. Analysis suggests that the model’s high perplexity scores are confounded by lexical frequency that was previously present in the pretrained dataset....
