ArXiv

grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP

2026-07-24 · 16:14 UTC ·Izzath Nisfer, Ashini Kavindya, Ovindu Atukorala... ·0 min read
Authors
Izzath Nisfer, Ashini Kavindya, Ovindu Atukorala...
Categories
cs.CL
arXiv
https://arxiv.org/abs/2607.22456v1
PDF
https://arxiv.org/pdf/2607.22456v1

Brief

grapheme-kit addresses errors introduced when metrics operate on Unicode code points rather than grapheme clusters by providing grapheme-level lexical distance, similarity, and evaluation functions in Python. It adds accurate cluster identification and composition/decomposition tools for Tamil and Sinhala, and an OCR case study shows grapheme-based metrics better reflect errors in complex scripts. Full paper text was not available here.

Why it matters

grapheme-kit is an open-source Python library (Izzath Nisfer et al., arXiv:2607.22456v1, 2026-07-24) that replaces Unicode codepoint–based lexical distance, similarity, and evaluation metrics with grapheme-cluster–level versions.

Key details

  • The library provides improved grapheme-cluster identification and composition/decomposition utilities for Tamil and Sinhala, and an OCR case study reported that grapheme-level metrics yield a more faithful evaluation of complex scripts (abstract only; full text not available here).
Source evidence

Abstract

Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points. We introduce grapheme-kit, an open-source Python library that extends these metrics to operate on grapheme clusters instead. The library also provides improved grapheme processing for Tamil and Sinhala, including accurate grapheme cluster identification and grapheme composition/decomposition utilities. Through an OCR case study, we demonstrate that grapheme-level metrics provide a more faithful evaluation of complex scripts.