Notes on Low-Resource Machine Learning and Morphological Tokenization

Low-resource natural language processing poses unique empirical challenges that standard large-scale pretraining recipes do not solve out of the box.

In agglutinative languages such as Mongolian, grammatical relationships (case, tense, possession) are expressed through successive suffix chains attached to root morphemes.

The Vocabulary Explosion Problem

Consider the root word ном (book). In an agglutinative system, valid derivations multiply rapidly:

• ном (book)
• номыг (book - accusative)
• номнууд (books - plural)
• номнуудыг (books - plural accusative)
• номноосоо (from one's own book - ablative reflexive)

A standard naive word-level tokenizer experiences rapid vocabulary explosion and out-of-vocabulary (OOV) tokens:

$$\text{OOV Rate} = \frac{|\{w \in \mathcal{D}_{\text{test}} \mid w \notin \mathcal{V}_{\text{train}}\}|}{|\mathcal{D}_{\text{test}}|}$$

Subword Segmentation: BPE vs. Unigram Language Models

Using SentencePiece with Byte-Pair Encoding (BPE), tokenization minimizes cross-entropy perplexity over subword units:

$$\mathcal{P} = \exp\left(-\frac{1}{N}\sum_{i=1}^N \log P(t_i \mid t_{<i})\right)$$

This retains stem-suffix boundaries, allowing sequence-to-sequence networks to generalize across unseen morphological combinations.