dinglehopper/qurator/dinglehopper/notebooks/Unicode normalization and Character segmentation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unicodedata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def list_characters(s):\n",
    "    \"\"\"List characters of string s, as seen by Python\"\"\"\n",
    "    for c in s:\n",
    "        print(c, end=' ')\n",
    "        if unicodedata.combining(c):\n",
    "            print(end=' ')\n",
    "        print(unicodedata.name(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing two Unicode strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S LATIN CAPITAL LETTER S\n",
      "c LATIN SMALL LETTER C\n",
      "h LATIN SMALL LETTER H\n",
      "l LATIN SMALL LETTER L\n",
      "y LATIN SMALL LETTER Y\n",
      "ñ LATIN SMALL LETTER N WITH TILDE\n",
      "\n",
      "S LATIN CAPITAL LETTER S\n",
      "c LATIN SMALL LETTER C\n",
      "h LATIN SMALL LETTER H\n",
      "l LATIN SMALL LETTER L\n",
      "y LATIN SMALL LETTER Y\n",
      "n LATIN SMALL LETTER N\n",
      "̃  COMBINING TILDE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "words = [unicodedata.normalize('NFC', 'Schlyñ'), unicodedata.normalize('NFD', 'Schlyñ')]\n",
    "\n",
    "for s in words:\n",
    "    list_characters(s)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two strings are different:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words[0] == words[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And yet they are the canonically equivalent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unicodedata.normalize('NFC', words[0]) == unicodedata.normalize('NFC', words[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "→ Normalize to NFC (Normalization Form Composed) to compare. NFC is also composed, which is what we want. But it doesn't matter because we're not interested in the characters as Python sees them, but in grapheme clusters (see below.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grapheme clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For evaluation we're interesting in what is perceived as \"characters\". But is \"ñ\" 1 character (LATIN SMALL LETTER N WITH TILDE) or 2 (LATIN SMALL LETTER N + COMBINING TILDE)?\n",
    "\n",
    "What we're probably want are [grapheme clusters](https://uniseg-python.readthedocs.io/en/latest/graphemecluster.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['S', 'c', 'h', 'l', 'y', 'ñ']\n",
      "['S', 'c', 'h', 'l', 'y', 'ñ']\n"
     ]
    }
   ],
   "source": [
    "from uniseg.graphemecluster import grapheme_clusters\n",
    "\n",
    "for w in words:\n",
    "    print(list(grapheme_clusters(w)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just looking at the interesting character – the last one - from both words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ñ LATIN SMALL LETTER N WITH TILDE\n",
      "\n",
      "n LATIN SMALL LETTER N\n",
      "̃  COMBINING TILDE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for w in words:\n",
    "    list_characters(list(grapheme_clusters(w))[-1])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "→ Work with grapheme clusters, not \"characters as Python sees them\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unicode_name(c):\n",
    "    if 0xE000 <= ord(c) <= 0xF8FF:\n",
    "        return 'private use character 0x{:04X}'.format(ord(c))\n",
    "    else:\n",
    "        return unicodedata.name(c)\n",
    " \n",
    "\n",
    "def list_grapheme_clusters(s):\n",
    "    \"\"\"List grapheme clusters of string s\"\"\"\n",
    "    for g in grapheme_clusters(s):\n",
    "        print(g, end=' ')\n",
    "        if len(g) > 1:\n",
    "            print('(multiple)', end=' ')\n",
    "        try:\n",
    "            print(', '.join(unicode_name(c) for c in g))\n",
    "        except ValueError:\n",
    "            print('ValueError')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S LATIN CAPITAL LETTER S\n",
      "c LATIN SMALL LETTER C\n",
      "h LATIN SMALL LETTER H\n",
      "l LATIN SMALL LETTER L\n",
      "y LATIN SMALL LETTER Y\n",
      "ñ LATIN SMALL LETTER N WITH TILDE\n",
      "\n",
      "S LATIN CAPITAL LETTER S\n",
      "c LATIN SMALL LETTER C\n",
      "h LATIN SMALL LETTER H\n",
      "l LATIN SMALL LETTER L\n",
      "y LATIN SMALL LETTER Y\n",
      "ñ (multiple) LATIN SMALL LETTER N, COMBINING TILDE\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for w in words:\n",
    "    list_grapheme_clusters(w)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "私 CJK UNIFIED IDEOGRAPH-79C1\n",
      "は HIRAGANA LETTER HA\n",
      "彼 CJK UNIFIED IDEOGRAPH-5F7C\n",
      "女 CJK UNIFIED IDEOGRAPH-5973\n",
      "が HIRAGANA LETTER GA\n",
      "お HIRAGANA LETTER O\n",
      "茶 CJK UNIFIED IDEOGRAPH-8336\n",
      "を HIRAGANA LETTER WO\n",
      "好 CJK UNIFIED IDEOGRAPH-597D\n",
      "き HIRAGANA LETTER KI\n",
      "な HIRAGANA LETTER NA\n",
      "事 CJK UNIFIED IDEOGRAPH-4E8B\n",
      "が HIRAGANA LETTER GA\n",
      "分 CJK UNIFIED IDEOGRAPH-5206\n",
      "か HIRAGANA LETTER KA\n",
      "っ HIRAGANA LETTER SMALL TU\n",
      "た HIRAGANA LETTER TA\n",
      "。 IDEOGRAPHIC FULL STOP\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('私は彼女がお茶を好きな事が分かった。')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ". FULL STOP\n",
      "  SPACE\n",
      "ا ARABIC LETTER ALEF\n",
      "م ARABIC LETTER MEEM\n",
      "ا ARABIC LETTER ALEF\n",
      "  SPACE\n",
      "چ ARABIC LETTER TCHEH\n",
      "ن ARABIC LETTER NOON\n",
      "د ARABIC LETTER DAL\n",
      "  SPACE\n",
      "ت ARABIC LETTER TEH\n",
      "ا ARABIC LETTER ALEF\n",
      "  SPACE\n",
      "ح ARABIC LETTER HAH\n",
      "ر ARABIC LETTER REH\n",
      "ف ARABIC LETTER FEH\n",
      "  SPACE\n",
      "ت ARABIC LETTER TEH\n",
      "و ARABIC LETTER WAW\n",
      "  SPACE\n",
      "ف ARABIC LETTER FEH\n",
      "ا ARABIC LETTER ALEF\n",
      "ر ARABIC LETTER REH\n",
      "س ARABIC LETTER SEEN\n",
      "ی ARABIC LETTER FARSI YEH\n",
      "  SPACE\n",
      "ه ARABIC LETTER HEH\n",
      "س ARABIC LETTER SEEN\n",
      "ت ARABIC LETTER TEH\n",
      "  SPACE\n",
      "ک ARABIC LETTER KEHEH\n",
      "ه ARABIC LETTER HEH\n",
      "  SPACE\n",
      "ت ARABIC LETTER TEH\n",
      "و ARABIC LETTER WAW\n",
      "  SPACE\n",
      "ع ARABIC LETTER AIN\n",
      "ر ARABIC LETTER REH\n",
      "ب ARABIC LETTER BEH\n",
      "ی ARABIC LETTER FARSI YEH\n",
      "  SPACE\n",
      "ن ARABIC LETTER NOON\n",
      "ی ARABIC LETTER FARSI YEH\n",
      "س ARABIC LETTER SEEN\n",
      "ت ARABIC LETTER TEH\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('. اما چند تا حرف تو فارسی هست که تو عربی نیست')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ". FULL STOP\n",
      "  SPACE\n",
      "ل ARABIC LETTER LAM\n",
      "ك ARABIC LETTER KAF\n",
      "ن ARABIC LETTER NOON\n",
      "  SPACE\n",
      "ك ARABIC LETTER KAF\n",
      "م ARABIC LETTER MEEM\n",
      "  SPACE\n",
      "ع ARABIC LETTER AIN\n",
      "د ARABIC LETTER DAL\n",
      "د ARABIC LETTER DAL\n",
      "  SPACE\n",
      "ا ARABIC LETTER ALEF\n",
      "ل ARABIC LETTER LAM\n",
      "ك ARABIC LETTER KAF\n",
      "ل ARABIC LETTER LAM\n",
      "م ARABIC LETTER MEEM\n",
      "ا ARABIC LETTER ALEF\n",
      "ت ARABIC LETTER TEH\n",
      "  SPACE\n",
      "ب ARABIC LETTER BEH\n",
      "ا ARABIC LETTER ALEF\n",
      "ل ARABIC LETTER LAM\n",
      "ف ARABIC LETTER FEH\n",
      "ا ARABIC LETTER ALEF\n",
      "ر ARABIC LETTER REH\n",
      "س ARABIC LETTER SEEN\n",
      "ي ARABIC LETTER YEH\n",
      "ة ARABIC LETTER TEH MARBUTA\n",
      "  SPACE\n",
      "ه ARABIC LETTER HEH\n",
      "ل ARABIC LETTER LAM\n",
      "  SPACE\n",
      "أ ARABIC LETTER ALEF WITH HAMZA ABOVE\n",
      "ن ARABIC LETTER NOON\n",
      "ت ARABIC LETTER TEH\n",
      "  SPACE\n",
      "ب ARABIC LETTER BEH\n",
      "ا ARABIC LETTER ALEF\n",
      "ل ARABIC LETTER LAM\n",
      "ل ARABIC LETTER LAM\n",
      "غ ARABIC LETTER GHAIN\n",
      "ة ARABIC LETTER TEH MARBUTA\n",
      "  SPACE\n",
      "ا ARABIC LETTER ALEF\n",
      "ل ARABIC LETTER LAM\n",
      "ع ARABIC LETTER AIN\n",
      "ر ARABIC LETTER REH\n",
      "ب ARABIC LETTER BEH\n",
      "ي ARABIC LETTER YEH\n",
      "ة ARABIC LETTER TEH MARBUTA\n",
      "؟ ARABIC QUESTION MARK\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('. لكن كم عدد الكلمات بالفارسية هل أنت باللغة العربية؟')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "H LATIN CAPITAL LETTER H\n",
      "e LATIN SMALL LETTER E\n",
      "l LATIN SMALL LETTER L\n",
      "l LATIN SMALL LETTER L\n",
      "😀 GRINNING FACE\n",
      "  SPACE\n",
      "W LATIN CAPITAL LETTER W\n",
      "😀 GRINNING FACE\n",
      "r LATIN SMALL LETTER R\n",
      "l LATIN SMALL LETTER L\n",
      "d LATIN SMALL LETTER D\n",
      "! EXCLAMATION MARK\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('Hell😀 W😀rld!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "u̶̜͓̬̞͚͙̪̰͓̯̲̝̬͔͎̳̼͇̓͊ͤ̋̃̀̄̓̿͊̀̚͟͜͟ͅ (multiple) LATIN SMALL LETTER U, COMBINING COMMA ABOVE, COMBINING NOT TILDE ABOVE, COMBINING LATIN SMALL LETTER E, COMBINING DOUBLE ACUTE ACCENT, COMBINING TILDE, COMBINING GRAVE ACCENT, COMBINING LEFT ANGLE ABOVE, COMBINING MACRON, COMBINING COMMA ABOVE, COMBINING DOUBLE OVERLINE, COMBINING NOT TILDE ABOVE, COMBINING DOUBLE MACRON BELOW, COMBINING GRAVE TONE MARK, COMBINING DOUBLE BREVE BELOW, COMBINING LONG STROKE OVERLAY, COMBINING DOUBLE MACRON BELOW, COMBINING LEFT HALF RING BELOW, COMBINING X BELOW, COMBINING CARON BELOW, COMBINING DOWN TACK BELOW, COMBINING DOUBLE RING BELOW, COMBINING ASTERISK BELOW, COMBINING BRIDGE BELOW, COMBINING TILDE BELOW, COMBINING X BELOW, COMBINING INVERTED BREVE BELOW, COMBINING LOW LINE, COMBINING UP TACK BELOW, COMBINING CARON BELOW, COMBINING LEFT ARROWHEAD BELOW, COMBINING UPWARDS ARROW BELOW, COMBINING DOUBLE LOW LINE, COMBINING SEAGULL BELOW, COMBINING EQUALS SIGN BELOW, COMBINING GREEK YPOGEGRAMMENI\n",
      "ņ̷͔̤̜̗̘̠̦̦̖̟͉̹͕̬͎̙̲̲̎̅̈́ͮͣ̔̀̌͂̄͆͑̚ (multiple) LATIN SMALL LETTER N, COMBINING DOUBLE VERTICAL LINE ABOVE, COMBINING OVERLINE, COMBINING GREEK DIALYTIKA TONOS, COMBINING LEFT ANGLE ABOVE, COMBINING LATIN SMALL LETTER V, COMBINING LATIN SMALL LETTER A, COMBINING REVERSED COMMA ABOVE, COMBINING GRAVE ACCENT, COMBINING CARON, COMBINING GREEK PERISPOMENI, COMBINING MACRON, COMBINING BRIDGE ABOVE, COMBINING LEFT HALF RING ABOVE, COMBINING SHORT SOLIDUS OVERLAY, COMBINING CEDILLA, COMBINING LEFT ARROWHEAD BELOW, COMBINING DIAERESIS BELOW, COMBINING LEFT HALF RING BELOW, COMBINING ACUTE ACCENT BELOW, COMBINING LEFT TACK BELOW, COMBINING MINUS SIGN BELOW, COMBINING COMMA BELOW, COMBINING COMMA BELOW, COMBINING GRAVE ACCENT BELOW, COMBINING PLUS SIGN BELOW, COMBINING LEFT ANGLE BELOW, COMBINING RIGHT HALF RING BELOW, COMBINING RIGHT ARROWHEAD BELOW, COMBINING CARON BELOW, COMBINING UPWARDS ARROW BELOW, COMBINING RIGHT TACK BELOW, COMBINING LOW LINE, COMBINING LOW LINE\n",
      "i̴̢͖̳̣̙͕̍ͯͧ̀ͥͭ̆ͣ̉͐͆̊͋͛̈́͒͟ (multiple) LATIN SMALL LETTER I, COMBINING VERTICAL LINE ABOVE, COMBINING LATIN SMALL LETTER X, COMBINING LATIN SMALL LETTER U, COMBINING GRAVE ACCENT, COMBINING LATIN SMALL LETTER I, COMBINING LATIN SMALL LETTER T, COMBINING BREVE, COMBINING LATIN SMALL LETTER A, COMBINING HOOK ABOVE, COMBINING RIGHT ARROWHEAD ABOVE, COMBINING BRIDGE ABOVE, COMBINING RING ABOVE, COMBINING HOMOTHETIC ABOVE, COMBINING ZIGZAG ABOVE, COMBINING GREEK DIALYTIKA TONOS, COMBINING FERMATA, COMBINING TILDE OVERLAY, COMBINING RETROFLEX HOOK BELOW, COMBINING DOUBLE MACRON BELOW, COMBINING RIGHT ARROWHEAD AND UP ARROWHEAD BELOW, COMBINING DOUBLE LOW LINE, COMBINING DOT BELOW, COMBINING RIGHT TACK BELOW, COMBINING RIGHT ARROWHEAD BELOW\n",
      "c̰̟̫̲͇̺̹͖̼̦̾ͮ̍̐ͤͪ̓ͤ̐̈́̅ͯͤ̚̚͘ (multiple) LATIN SMALL LETTER C, COMBINING VERTICAL TILDE, COMBINING LATIN SMALL LETTER V, COMBINING VERTICAL LINE ABOVE, COMBINING CANDRABINDU, COMBINING LATIN SMALL LETTER E, COMBINING LEFT ANGLE ABOVE, COMBINING LATIN SMALL LETTER H, COMBINING COMMA ABOVE, COMBINING LATIN SMALL LETTER E, COMBINING LEFT ANGLE ABOVE, COMBINING CANDRABINDU, COMBINING GREEK DIALYTIKA TONOS, COMBINING OVERLINE, COMBINING LATIN SMALL LETTER X, COMBINING LATIN SMALL LETTER E, COMBINING DOT ABOVE RIGHT, COMBINING TILDE BELOW, COMBINING PLUS SIGN BELOW, COMBINING INVERTED DOUBLE ARCH BELOW, COMBINING LOW LINE, COMBINING EQUALS SIGN BELOW, COMBINING INVERTED BRIDGE BELOW, COMBINING RIGHT HALF RING BELOW, COMBINING RIGHT ARROWHEAD AND UP ARROWHEAD BELOW, COMBINING SEAGULL BELOW, COMBINING COMMA BELOW\n",
      "o̴ͣ̑̐ͫ̈̄͊ͥ̓͟͏̫͔̠̤̜̤̥͘ (multiple) LATIN SMALL LETTER O, COMBINING LATIN SMALL LETTER A, COMBINING INVERTED BREVE, COMBINING CANDRABINDU, COMBINING LATIN SMALL LETTER M, COMBINING DIAERESIS, COMBINING MACRON, COMBINING NOT TILDE ABOVE, COMBINING LATIN SMALL LETTER I, COMBINING GREEK KORONIS, COMBINING DOUBLE MACRON BELOW, COMBINING TILDE OVERLAY, COMBINING GRAPHEME JOINER, COMBINING DOT ABOVE RIGHT, COMBINING INVERTED DOUBLE ARCH BELOW, COMBINING LEFT ARROWHEAD BELOW, COMBINING MINUS SIGN BELOW, COMBINING DIAERESIS BELOW, COMBINING LEFT HALF RING BELOW, COMBINING DIAERESIS BELOW, COMBINING RING BELOW\n",
      "ḍ̛̥͖͓̪͈̹̯͖̱̘͙͖ͧ̿ͧ̓̓͊̈͑͘̕ (multiple) LATIN SMALL LETTER D, COMBINING LATIN SMALL LETTER U, COMBINING DOUBLE OVERLINE, COMBINING LATIN SMALL LETTER U, COMBINING COMMA ABOVE, COMBINING COMMA ABOVE, COMBINING NOT TILDE ABOVE, COMBINING DIAERESIS, COMBINING LEFT HALF RING ABOVE, COMBINING DOT ABOVE RIGHT, COMBINING COMMA ABOVE RIGHT, COMBINING HORN, COMBINING DOT BELOW, COMBINING RING BELOW, COMBINING RIGHT ARROWHEAD AND UP ARROWHEAD BELOW, COMBINING X BELOW, COMBINING BRIDGE BELOW, COMBINING DOUBLE VERTICAL LINE BELOW, COMBINING RIGHT HALF RING BELOW, COMBINING INVERTED BREVE BELOW, COMBINING RIGHT ARROWHEAD AND UP ARROWHEAD BELOW, COMBINING MACRON BELOW, COMBINING LEFT TACK BELOW, COMBINING ASTERISK BELOW, COMBINING RIGHT ARROWHEAD AND UP ARROWHEAD BELOW\n",
      "e̛̺͈̜̰̜̖͎͚͈͋̒̆̈́̏͊ͬ̎̑̇̾̆̓ͬ̔̐̾ͭ́͞ (multiple) LATIN SMALL LETTER E, COMBINING HOMOTHETIC ABOVE, COMBINING TURNED COMMA ABOVE, COMBINING BREVE, COMBINING GREEK DIALYTIKA TONOS, COMBINING DOUBLE GRAVE ACCENT, COMBINING NOT TILDE ABOVE, COMBINING LATIN SMALL LETTER R, COMBINING DOUBLE VERTICAL LINE ABOVE, COMBINING INVERTED BREVE, COMBINING DOT ABOVE, COMBINING VERTICAL TILDE, COMBINING BREVE, COMBINING GREEK KORONIS, COMBINING LATIN SMALL LETTER R, COMBINING REVERSED COMMA ABOVE, COMBINING CANDRABINDU, COMBINING VERTICAL TILDE, COMBINING LATIN SMALL LETTER T, COMBINING ACUTE TONE MARK, COMBINING HORN, COMBINING DOUBLE MACRON, COMBINING INVERTED BRIDGE BELOW, COMBINING DOUBLE VERTICAL LINE BELOW, COMBINING LEFT HALF RING BELOW, COMBINING TILDE BELOW, COMBINING LEFT HALF RING BELOW, COMBINING GRAVE ACCENT BELOW, COMBINING UPWARDS ARROW BELOW, COMBINING DOUBLE RING BELOW, COMBINING DOUBLE VERTICAL LINE BELOW\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('u̶̜͓̬̞͚͙̪̰͓̯̲̝̬͔͎̳̼͇̓͊ͤ̋̃̀̄̓̿͊̀̚͟͜͟ͅņ̷͔̤̜̗̘̠̦̦̖̟͉̹͕̬͎̙̲̲̎̅̈́ͮͣ̔̀̌͂̄͆͑̚i̴̢͖̳̣̙͕̍ͯͧ̀ͥͭ̆ͣ̉͐͆̊͋͛̈́͒͟c̰̟̫̲͇̺̹͖̼̦̾ͮ̍̐ͤͪ̓ͤ̐̈́̅ͯͤ̚̚͘o̴ͣ̑̐ͫ̈̄͊ͥ̓͟͏̫͔̠̤̜̤̥͘ḍ̛̥͖͓̪͈̹̯͖̱̘͙͖ͧ̿ͧ̓̓͊̈͑͘̕e̛̺͈̜̰̜̖͎͚͈͋̒̆̈́̏͊ͬ̎̑̇̾̆̓ͬ̔̐̾ͭ́͞')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Z LATIN CAPITAL LETTER Z\n",
      "e LATIN SMALL LETTER E\n",
      "u LATIN SMALL LETTER U\n",
      "g LATIN SMALL LETTER G\n",
      "n LATIN SMALL LETTER N\n",
      "uͤ (multiple) LATIN SMALL LETTER U, COMBINING LATIN SMALL LETTER E\n",
      "ß LATIN SMALL LETTER SHARP S\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('Zeugnuͤß')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Z LATIN CAPITAL LETTER Z\n",
      "e LATIN SMALL LETTER E\n",
      "u LATIN SMALL LETTER U\n",
      "g LATIN SMALL LETTER G\n",
      "n LATIN SMALL LETTER N\n",
      " private use character 0xE72B\n",
      "ß LATIN SMALL LETTER SHARP S\n"
     ]
    }
   ],
   "source": [
    "list_grapheme_clusters('Zeugnß')"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}