How Should We Model the Probability of a Language?
Type de ressource
Manuscript
Auteurs/contributeurs
- Dent, Rasul (Author)
- Suarez, Pedro Ortiz (Author)
- Clérice, Thibault (Author)
- Sagot, Benoît (Author)
Title
How Should We Model the Probability of a Language?
Abstract
Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.
Date
2026-02
Accessed
25/08/2026 09:04
Library Catalog
HAL
Notes
working paper or preprint
Référence
Dent, R., Suarez, P. O., Clérice, T., & Sagot, B. (2026). How Should We Model the Probability of a Language? https://inria.hal.science/hal-05513952
Type de papier
Lien vers cette notice