Emoji under the hood
For the past few weeks, Iโve been implementing emoji support for Skija. I thought it might be fun sharing a few nitty-gritty details of how this โbiggest innovation in human communication since the invention of the letter ๐ ฐ๏ธโ works under the hood.
Warning: some emoji might not display as expected on your device. In that case, use this text version:
Intro to Unicode
As you might know, all text inside computers is encoded with numbers. One letterโone number. The most popular encoding we use is called Unicode, with the two most popular variations called UTF-8 and UTF-16.
Unicode allocates 221 (~2 mil) characters called codepoints. Sorry, programmers, 21 is not a multiple of 8 ๐คท. Out of those 2 million, actually defined are ~150k characters.
150k defined characters cover all the scripts used on ๐, many dead languages, a lot of weird stuff like ๐ฃ๐ฒ๐ซ๐ซ๐ถ ๐๐๐ฅ๐ฅ๐๐ฃ๐ค, sษนวสสวl uสop-วpแดsdn, GHz as one glyph: ใ, โrightwards two-headed arrow with tail with double vertical strokeโ: โค, seven-eyed monster: ๊ฎ, and a duck:
As a side note, definitely check out the Egyptian Hieroglyphs block (U+13000โU+1342F). They have some really weird stuff.
Basic emoji
So, emoji. At their simplest, they are just that: another symbol in a Unicode table. Most of them are grouped in U+1F300โ1F6FF and U+1F900โ1FAFF.
Thatโs why emoji behave like any other letter: they can be typed in a text field, copied, pasted, rendered in a plain text document, embedded in a tweet, etc. When you type โAโ, the computer sees U+0041. When you type โ๐ตโ, the computer sees U+1F335. Not much difference.
Emoji fonts
Why are emoji rendered as images then? Well, bitmap fonts. Apparently, you can create a font that has pngs for glyphs instead of boring black-and-white vector shapes.
Every OS comes with a single pre-installed font for emoji. On macOS/iOS, thatโs Apple Color Emoji. Windows has Segoe UI Emoji, Android has Noto Color Emoji.
As far as I can tell, Apple is a bitmap font with 160ร160 raster glyphs, Noto uses 128ร128 bitmaps, and Segoe is a vector color font ๐.
Thatโs why emoji look different on different devicesโjust like fonts look different! On top of that, many apps bundle their own emoji fonts, too: WhatsApp, Twitter, Facebook.
Font fallbacks
Now about the rendering. You donโt write your text in Apple Color Emoji or Segoe UI Emoji fonts (unless you are really young and pure at heart โค๏ธ). So how can a text set in e.g. Helvetica include emoji?
Well, with the same machinery that makes Cyrillic text look ugly in Clubhouse or on Medium: font fallbacks.
When you type, say, U+1F419, it is first looked up in your current font. Letโs say itโs San Francisco. San Francisco doesnโt have a glyph for U+1F419, so OS starts to look for any other installed font that might have it.
U+1F419 can only be found in Apple Color Emoji, thus OS uses it to render U+1F419 (rest of the text stays in your current font). In the end, you see ๐. Thatโs why, no matter which font you use, Emoji always look the same:
Variation selector-16
Not all Emoji started their life straight in Emoji code block. In fact, pictograms existed in fonts and Unicode at least since 1993. Look in Miscellaneous Symbols U+2600-26FF and Dingbats U+2700-27FF:
These glyphs are as normal as any other letters we use: they are single-codepoint, black-and-white, and many fonts have them included. For example, here are all the differnt fonts on my machine that have their own version of โ๏ธ (U+2702 BLACK SCISSORS):
Guess what? When Apple Color Emoji was created, it had its own version of the same U+2702 codepoint that looked like this:
Now for the tricky part. How does OS knows when to render โ๏ธ and when โ๏ธ, if both of them have the same codepoint and not only Apple Color Emoji has it, but also many other higher-priority traditional fonts?
Meet U+FE0F, also known as VARIATION SELECTOR-16. Itโs a hint to the text renderer to switch to an emoji font.
U+2702 โ โ๏ธ U+2702 U+FE0F โ โ๏ธ U+2697 โ โ๏ธ U+2697 U+FE0F โ โ๏ธ U+26A0 โ โ๏ธ U+26A0 U+FE0F โ โ๏ธ U+2618 โ โ๏ธ U+2618 U+FE0F โ โ๏ธ
Simple, elegant, and no need to allocate new codepoints while the old ones are already there. After all, things like โ ๏ธ and โ ๏ธ have the same meaning, only the presentation is different.
Grapheme clusters
Here we encounter another problem โ our emoji are now not a single codepoint, but two. This means we need a way to define character boundaries.
Meet Grapheme Clusters. Grapheme cluster is a sequence of codepoints that is considered a single human-perceived glyph.
Grapheme Clusters were not invented just for emoji, they apply to normal alphabets too. โUฬโ is a single grapheme cluster, even though itโs composed of two codepoints: U+0055 UPPER-CASE U followed by U+0308 COMBINING DIAERESIS.
Grapheme clusters create many complications for programmers. You canโt just do substring(0, 10) to take the first 10 charactersโyou might split an emoji in half (or an acute, so donโt do it anyway)!
Reversing a string is tricky, tooโwhile U+263A U+FE0F makes sense, U+FE0F U+263A does not.
Finally, you canโt just call .length on a string. Well, you can, but the result will surprise you. If you a developer, try this "๐คฆ๐ผโโ๏ธ".length in your browserโs console.
A tip for programmers: if you are working with text, get a library that is grapheme clusters-aware. For C, C++m and JVM that would be ICU, Swift does the right thing out-of-the-box, for others, see for yourself.
Grapheme clusters awareness month, anyone? Graphemes donโt want to be split! Oh, who am I kidding? for (int i = 0; i < str.length; ++i) str[i] go brrr!
Oh, by the way, did I mentioned that this: ลฒฬทฬกฬกฬจฬซอฬฬฏฬฃออฬฬฑฬฬฑฬฃออฬซอฬฎฬซฬนฬฬฃอฬฆฬฌฬฌอออออฬฉฬฌฬฬฬฬฬฬพอฬอออฬฬพฬฬฬฬฬ ฬอฬฬอฬฬฬอ อ is a single grapheme cluster, too? Its length is 65, and it shouldnโt ever be split in half. Sleep tight ๐ :)
Skin tone modifier
Most human Emoji depict an abstract yellow person. When skin tone was added in 2015, instead of adding a new codepoint for each emoji and skin tone combination, only five new codepoints were added: ๐ป๐ผ๐ฝ๐พ๐ฟ U+1F3FB..U+1F3FF.
These are not supposed to be used on their own but to be appended to the existing emoji. Together they form a ligature: ๐ (U+1F44B WAVING HAND SIGN) directly followed by ๐ฝ (U+1F3FD MEDIUM SKIN TONE MODIFIER) becomes ๐๐ฝ.
๐๐ฝ does not have its own codepoint (itโs a sequence of two: U+1F44B U+1F3FD), but it has its own unique look. With just five modifiers, ~280 human emojis got turned into 1680 variations. Hereโre some dancers:
๐บ๐บ๐ป๐บ๐ผ๐บ๐ฝ๐บ๐พ๐บ๐ฟ
Zero-width Joiner
Letโs say your friend just sent you a picture of an apple she is growing in her garden. You need to replyโhow? You might send a ๐ฉ WOMAN EMOJI (U+1F469), followed by a ๐พ SHEAF OF RICE (U+1F33E). If you put the two together: ๐ฉ๐พ, nothing happens. Itโs just two separate emoji.
But! If you add U+200D in between, magic happens: they turn into the one ๐ฉโ๐พ woman farmer.
U+200D is called ZERO-WIDTH JOINER, or ZWJ for short. It works similarly to what we saw with skin tone, but this time you can join two self-sufficient emoji into one. Not all combinations work, but many do, sometimes in surprising ways!
Some examples:
๐ฉ + โ๏ธ โ ๐ฉโโ๏ธ ๐จ + ๐ป โ ๐จโ๐ป ๐ฐ + โ๏ธ โ ๐ฐโโ๏ธ ๐ป + โ๏ธ โ ๐ปโโ๏ธ ๐ด + โ ๏ธ โ ๐ดโโ ๏ธ ๐ณ๏ธ + ๐ โ ๐ณ๏ธโ๐
One weird inconsistency Iโve noticed is that hair color is done via ZWJ, while skin tone is just modifier emoji with no joiner. Why? Seriously, I am asking you: why? I have no clue.
๐จ + ๐ฟ U+1F3FF โ ๐จ๐ฟ ๐จ + ZWJ + ๐ฆฐ โ ๐จโ๐ฆฐ
Unfortunately, some emoji are NOT implemented as combinations with ZWJ. I consider those missing opportunities:
๐จ + ๐ฆท โ ๐ง ๐จ + ๐ โ ๐ง ๐ฉ + ๐ โ ๐ต๏ธโโ๏ธ ๐ + ๐ โ ๐ ๐ + ๐ โ ๐ ๐ + ๐ง โ โ๏ธ ๐ด + ๐ โ ๐ฆ ๐ + ๐ โ ๐ฃ ๐ + ๐ฆ โ ๐ ๐ฆต + ๐ฆต + ๐ช + ๐ช + ๐ + ๐ + ๐ + ๐ + ๐ + ๐ง โ ๐ง
How do you type ZWJ? You donโt. But you can copy it from here: โโ. Note: this is a special character, so expect it to behave weird. Itโs invisible, too. But itโs there.
Another big area where ZWJ shines is families and relationships configuration. A short story to illustrate:
๐จ๐ป + ๐ค + ๐จ๐ผ โ ๐จ๐ปโ๐คโ๐จ๐ผ ๐จ + โค๏ธ + ๐จ โ ๐จโโค๏ธโ๐จ ๐จ + โค๏ธ + ๐ + ๐จ โ ๐จโโค๏ธโ๐โ๐จ ๐จ + ๐จ + ๐ง โ ๐จโ๐จโ๐ง ๐จ + ๐จ + ๐ง + ๐ง โ ๐จโ๐จโ๐งโ๐ง
Flags
Country flags are part of the Unicode standard, but for some reason are not implemented on Windows. If you are reading this in a browser from WindowsโI am sorry!
Flags donโt have dedicated codepoints. Instead, they are two-letter ligatures.
๐บ + ๐ณ = ๐บ๐ณ ๐ท + ๐บ = ๐ท๐บ ๐ฎ + ๐ธ = ๐ฎ๐ธ ๐ฟ + ๐ฆ = ๐ฟ๐ฆ ๐ฏ + ๐ต = ๐ฏ๐ต
They donโt use real letters, though. Instead, the โregional indicator symbol letterโ alphabet is used (U+1F1E6..1F1FF). These letters are not used for anything but composing flags.
What happens if you put together two random letters? Not much: ๐ฝ๐พ (except that text editing starts to behave strange).
If you want to experiment, feel free to copy and combine from this alphabet: ๐ฆ ๐ง ๐จ ๐ฉ ๐ช ๐ซ ๐ฌ ๐ญ ๐ฎ ๐ฏ ๐ฐ ๐ฑ ๐ฒ ๐ณ ๐ด ๐ต ๐ถ ๐ท ๐ธ ๐น ๐บ ๐ป ๐ผ ๐ฝ ๐พ ๐ฟ. There are 258 valid two-letter combinations. Can you find them all?
A funny side-effect of being two-letter ligature: ''.join(reversed('๐บ๐ฆ')) => '๐ฆ๐บ'
Tag Sequences
Two-letter ligatures are cool, but donโt you want to be cooler? How about 32-letter ligatures? Meet tag sequences.
Tag sequence is a sequence of normal emoji, followed by another flavor of Latin letters (U+E0020..E007E), terminated with U+E007F CANCEL TAG.
Currently they are used for these three flags only: England, Scotland and Wales:
๐ด + gbeng + E007F = ๐ด๓ ง๓ ข๓ ฅ๓ ฎ๓ ง๓ ฟ
๐ด + gbsct + E007F = ๐ด๓ ง๓ ข๓ ณ๓ ฃ๓ ด๓ ฟ
๐ด + gbwls + E007F = ๐ด๓ ง๓ ข๓ ท๓ ฌ๓ ณ๓ ฟKeycaps
Not super-exciting, but needed for completeness: keycap sequences use yet another convention.
It goes like this: take a digit, * or #, turn it into emoji with U+FE0F, wrap into a square with U+20E3 COMBINING ENCLOSING KEYCAP:
* + FE0F + 20E3 = *๏ธโฃ
In total there are only twelve of them:
#๏ธโฃ *๏ธโฃ 0๏ธโฃ 1๏ธโฃ 2๏ธโฃ 3๏ธโฃ 4๏ธโฃ 5๏ธโฃ 6๏ธโฃ 7๏ธโฃ 8๏ธโฃ 9๏ธโฃ
Unicode updates
Unicode is updated every year, and emoji are a major part of each release. E.g. in Unicode 13 (March 2020), 55 new Emoji were added.
At the moment of writing neither the latest macOS (11.2.3) nor iOS (14.4.1) support emoji from Unicode 13 like
๐ฎโ๐จ, โค๏ธโ๐ฅ, ๐งโโ or ๐จ๐ปโโค๏ธโ๐โ๐จ๐ผ
For future generations, this is what I see in March 2021:
But, thanks to the magic of ZWJ, I can still figure out whatโs going on, just not in the most optimal way.
Conclusion
To sum up, these are seven ways emoji can be encoded:
- A single codepoint
๐ง U+1F9DB - Single codepoint + variation selector-16
โน๏ธ U+2639 + U+FE0F = โน๏ธ - Skin tone modifier
๐คต U+1F935 + U+1F3FD = ๐คต๐ฝ - Zero-width joiner sequence
๐จ + ZWJ + ๐ญ = ๐จโ๐ญ - Flags
๐ฆ + ๐ฑ = ๐ฆ๐ฑ - Tag sequences
๐ด + gbsct + U+E007F = ๐ด๓ ง๓ ข๓ ณ๓ ฃ๓ ด๓ ฟ - Keycap sequences
* + U+FE0F + U+20E3 = *๏ธโฃ
Techniques from 1-4 can be combined to construct a pretty complex message:
U+1F6B5 ๐ต Person Mountain Biking + U+1F3FB Light Skin Tone + U+200D ZWJ + U+2640 โ๏ธFemale Sign + U+FE0F Variation selector-16 = ๐ต๐ปโโ๏ธ Woman Mountain Biking: Light Skin Tone
If you are a programmer, remember to always use the ICU library to:
- extract a substring,
- measure string length,
- reverse a string.
The keyword to google is โGrapheme Clusterโ. It applies to emoji, to diacritics in Western languages, to Indic and Korean scripts, so please be aware.
Thatโs all I have. I hope the deeper understanding of how emoji work under the hood will help you in your work... Nah, just kidding. Hope you enjoyed it, though โ๏ธ
