Mongolian Image Captioning with Deep Vision Encoders

Image captioning requires an algorithmic system to parse spatial semantic features from an input raster image $I$ and generate a syntactically and semantically coherent natural language sentence $S = (w_1, w_2, \dots, w_T)$.

For widely spoken languages like English, vast multi-modal datasets like MS-COCO provide millions of captions. For low-resource languages such as Mongolian, parallel image-caption corpora are scarce.

Formulation and Objective Function

The caption generation is trained under a maximum likelihood objective:

Where:

• $\mathbf{v}_I \in \mathbb{R}^D$ is the deep visual feature representation extracted by the convolutional backbone (ResNet-50 / ResNet-101).
• $w_{<t} = (w_1, \dots, w_{t-1})$ represents the sequence of tokens generated prior to step $t$.
• $\theta$ parameterizes the sequence decoder (LSTM with soft attention mechanisms).

import torch
import torch.nn as nn
import torchvision.models as models

class ImageCaptioningModel(nn.Module):
    def __init__(self, vocab_size: int, embed_dim: int, hidden_dim: int):
        super().__init__()
        # Pretrained visual feature extractor
        resnet = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
        modules = list(resnet.children())[:-1]
        self.encoder = nn.Sequential(*modules)
        self.linear_proj = nn.Linear(resnet.fc.in_features, embed_dim)
        
        # Word embedding and recurrent language decoder
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.decoder = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc_out = nn.Linear(hidden_dim, vocab_size)

    def forward(self, images: torch.Tensor, captions: torch.Tensor) -> torch.Tensor:
        with torch.no_grad():
            features = self.encoder(images)
        features = features.reshape(features.size(0), -1)
        features = self.linear_proj(features).unsqueeze(1)
        
        embeddings = self.embedding(captions)
        inputs = torch.cat((features, embeddings), dim=1)
        hiddens, _ = self.decoder(inputs)
        outputs = self.fc_out(hiddens)
        return outputs

Attention Weights & Low-Resource Adaptation

To overcome morphological sparsity in Mongolian, words were tokenized with Byte-Pair Encoding (BPE) to preserve grammatical inflections. The visual attention map computes scalar alignment energies:

The normalized context vector $\mathbf{c}_t$ dynamically weights image regions during generation:

Interactive Waveform Representation

Below is an interactive harmonic field representing dynamic state representations: