URL slugs that keep the original script.

slugify/2 turns a title into a slug in its own writing system: Persian stays Persian, Hindi stays Hindi, Japanese stays Japanese. Only Latin letters change: they lose their accents the way WordPress writes them (Café gives cafe), unless latin: :keep.

iex> MishkaSlug.slugify("قیمت‌گذاری نرم‌افزار")
{:ok, "قیمتگذاری-نرمافزار"}

iex> MishkaSlug.slugify("हिन्दी भाषा")
{:ok, "हिन्दी-भाषा"}

iex> MishkaSlug.slugify("😀🚀")
{:error, :empty}

What a slug is made of

A slug is lowercase words joined by -. A word is a run of letters and decimal digits, with the combining marks that complete them (Hindi vowel signs and viramas, Thai tone marks). Everything else, from spaces and punctuation to emoji and / ? # % \ :, separates words. The result is NFC, so a stored slug compares equal to a pasted one (Hangul jamo are composed even after a prepended sign such as Tulu-Tigalari U+113D1, which OTP's normalizer leaves decomposed). Latin letters lose their accents the way WordPress writes them (Café Crème gives cafe-creme, see :latin), an apostrophe inside a word is removed (don't gives dont), _ stays in a word (snake_case gives snake_case, see :keep and :trim_kept), and ., & and , separate words (Q&A gives q-a, 1,000 gives 1-000). A capital sigma that ends a Greek word becomes ς (ΟΔΟΣ gives οδος) in every language but tr and az.

Safety

These are always on:

• Invisible characters are removed, never turned into a break: zero-width spaces and joiners, bidi controls (U+202E and the isolates), soft hyphens, byte-order marks, variation selectors, controls, private-use and unassigned code points. So pay\u200Bpal gives paypal.
• A word may not mix scripts (see :mixed_scripts).
• The slug can never hold a URL delimiter or be . or .., with or without :trim_kept.
• Arabic harakat, tatweel and Hebrew points are dropped (they are optional spelling); every other mark is kept, because in Indic and Thai scripts the marks spell the word.

Idempotent

slugify(slug, opts) returns {:ok, slug} for every slug it made with the same options. A CMS can therefore re-slug an incoming URL parameter: when the result differs, redirect to it.

Options

Options are checked by MishkaSlug.Options: an unknown option, or a value outside the accepted ones, raises ArgumentError.

• :language - String.t() | nil, default nil. The language of the text, such as a page's language ("fa", "tr"). It picks the lowercasing rules (tr and az turn I into ı), for fa, ur and ckb, folds the Arabic ي ى ك into the Persian ی ک, and for de, da, ca, sr and bs picks how latin: :ascii spells their letters.

• :locales - [String.t()], default []. The site's languages (["fa", "en"]). Only letters and digits of their scripts, and Latin, are kept; see MishkaSlug.Scripts. [] keeps every script. Without a :language, a Persian-family locale without ar turns on the Persian fold. A script subtag picks the script ("sr-Latn" keeps Latin only); an unknown language is {:error, {:unknown_locale, code}} unless a :script_resolver knows it.

• :script_resolver - {module, function, args} | nil, default nil. Decides the scripts of each :locales code instead of the built-in table: called as apply(module, function, [code | args]), it returns {:ok, [script]} (atoms such as :arabic, :latin, :cyrillic, :han; MishkaSlug.Scripts.from_iso15924/1 turns a CLDR code like "Arab" into them) or :error, which gives {:error, {:unknown_locale, code}}. nil uses the :script_resolver of the :mishka_slug application config, and without one the built-in table.

• :mixed_scripts - :split | :reject, default :split. What happens to a word that changes script, like pаypal with a Cyrillic а: :split cuts it at the change, :reject returns {:error, :mixed_scripts}.

• :zwnj - :remove | :separator, default :remove. The Persian half-space (U+200C): :remove glues the parts (قیمتگذاری), :separator makes it a word break (قیمت-گذاری).

• :digits - :keep | :ascii, default :keep. :keep leaves digits as typed (۱۴۰۳); :ascii writes every decimal digit as 0-9 (1403). With :keep, a digit of a script outside :locales is dropped like a letter; :ascii writes it as 0-9 first, so it stays.

• :latin - :ascii | :keep, default :ascii. What happens to accented Latin letters. :ascii writes them as WordPress's remove_accents does: a letter is decomposed and its marks dropped (é gives e, Vietnamese ệ gives e), and the letters that do not decompose get a plain spelling (ß gives s, æ ae, ø o, þ th, ł l, ı i). The :language adds its own spelling: de writes ä ö ü ß as ae oe ue ss, da writes æ ø å as ae oe aa, ca joins l·l into ll, and sr and bs write đ as dj. Only Latin letters change: Greek, Cyrillic, Arabic and every other script keep their marks. :keep leaves Latin letters as typed (Café gives café).

• :keep - list of ".", "_" and "~", default ["_"]. Characters kept inside a word (["."] keeps v1.2). Only the URL-safe ., _ and ~ are accepted. By default _ is kept like WordPress and Django keep it (snake_case_name); keep: [] makes it a word break again (snake-case-name). The list replaces the default, so keep: ["."] breaks on _; write [".", "_"] to keep both. See :trim_kept for the ends of a word.

• :trim_kept - boolean(), default false. Whether kept characters at the start or end of a word are removed. false leaves them like WordPress does (__init__ gives __init__, my_post_ gives my_post_, _ _ _ gives _-_-_), so a migrated WordPress slug stays the same. true trims them (__init__ gives init, my_post_ gives my_post, _ _ _ gives {:error, :empty}). A kept character that a prepended sign glues into one grapheme is removed too, and the sign stays (Malayalam ൎ. gives ൎ). An edge - is always removed, and a slug that is only . or .. is refused ({:error, :empty}) in both modes.

• :max_length - pos_integer() | nil, default nil. The longest slug, in code points (what Postgres and Ash count). Whole words are kept while they fit; a first word that is too long is cut between two graphemes.