At the recent FileMaker Developer Conference I was doing one of my fmSMS demonstrations to an attendee which involved me sending a message from fmSMS to their mobile phone (showing how you can use FileMaker to send SMS messages) and then having them send a reply back to demonstrate how you can use fmSMS to receive incoming messages directly into FileMaker. Normally the reply is a simple text reply of a few words, but this one was a bit different as you can see here:
I was pleasantly surprised to see that the Emoji reply had made it all the way back from their phone to fmSMS via the SMS Gateway and the PHP page that is used to convert the incoming reply into a FileMaker record without me having to do anything to handle the Unicode characters – it just worked! I decided to do a bit more research into this to see how different SMS Gateways handled Emoji characters as a way of testing their Unicode support (there are now hundreds of Emoji characters encoded in the Unicode standards).
It’s important here to understand the history of SMS – back when the GSM standard was being adopted the mobile phone industry decided on a standard set of characters called GSM 03.38. Support for this character set became mandatory for all GSM handsets and network elements (carriers etc). The GSM character set includes the English alphabet ( A-Z ), numbers (0 – 9) and some special characters, and the size of a single SMS was limited to 160 characters. The 160 character maximum actually comes from the fact that you can encode 160 7-bit characters into 140 bytes – 140 bytes being the limit for the size of a message.
Unicode characters use several GSM characters to describe each Unicode character which means that you won’t be able to send as many characters in your SMS when include Unicode characters. Depending on which special characters you’re sending, you may only be able to send between 35 and 70 characters. To send a message that is longer than the 160 characters/140 bytes limit the message needs to use Concatenation, which involves breaking up the message over multiple SMS messages.
Thankfully most SMS Gateways and mobile phone handsets support concatenation where multiple messages are joined together to form a single message on the handset, even though that single message is greater than 160 characters (or 140 bytes). As each message needs to broken up into individual 140 byte messages the SMS Gateways will charge you for each individual message, even though they appear as a single message to the recipient.
Here’s an example of such a message using the GSM character set which is 312 characters but appears as a single received message on the phone:
SMS Gateways that handle the encoding of Unicode characters make life easy for us developers – we don’t have to do anything when sending or receiving messages from FileMaker. One such SMS Gateway is Twilio which I used in my tests and was able to handle the Emoji characters with ease – they have a number of articles on their website that go into the details of their support for Unicode. I created an outgoing message in fmSMS with some Emoji characters (FileMaker Pro 7 and later being a fully Unicode aware application):
and sent the message using a number of different SMS Gateways. Where the SMS Gateway supported Unicode the message appeared exactly as it was sent:
I tried with some SMS Gateways that don’t have native support for Unicode characters and on the handset here’s what the same message looked like:
I then sent some replies containing Emoji characters back to fmSMS which has a “chat view” that shows all the sent/received messages to an individual contact – here’s our Emoji conversation:
As you can see the incoming message containing Emoji characters was received successfully using the Twilio SMS Gateway once again.
Unicode support isn’t only important for Emojis – if you need to send accented characters or messages in Greek, Cyrillic, Hebrew, Arabic, Thai, Chinese, Japanese, Korean etc then you will be thankful for Unicode support as well. Just remember that when sending messages containing Unicode characters the standard 160 character test doesn’t apply – I’m yet to find a way to have a Unicode character check that converts back to the GSM character set in FileMaker so I can accurately count the number of credits required for each message, regardless of whether it contains GMS or Unicode characters). If anyone knows of a way please let me know in the comments below. 😃
