Character-level recurrent neural networks are attractive for modelling text specifically because of their low input and output dimensionality. You have only so many chars to represent - lowercase letters, uppercase letters, digits and various auxillary characters, so you end up with 50-100 dimensions (each char is represented in one-hot encoding).
Still, it’s a drag to model upper and lower case separately. It adds to dimensionality, and perhaps more importantly, a network gets no clue that ‘a’ and ‘A’ actually represent pretty much the same thing.
The simplest solution is to discard uppercase and just use lowercase. We propose a more elegant way to deal with the two problems mentioned above: inserting special markers before each uppercase letter.
Hello World -> ^hello ^world
The resulting text is still quite readable. Of course you need to make sure there are no carets in your input to start with, but this is a minor matter: one could use any character as a marker, or invent one. Remember that a char is just a sparse vector: we can make it longer by one element, and that abstract element can be our marker.
Gents witnessing the emergence of R33, the very first char-RNN, back in the day
Here’s how to convert mixed-case text:
s = 'Hello World' re.sub( '([A-Z])', '^\\1', s ).lower()
What we do is insert a caret befor each uppercase letter and then turn the whole string to lowercase (
\1 is a backreference to a subgroup marked by parens in the first pattern; we need to quote the backslash, hence
\\1). An alternative is to perform both operations inside
sub() using a function to modify the match and return a replacement:
re.sub( '([A-Z])', lambda match: "^" + match.group( 1 ).lower(), s )
Should we need to convert stuff back, we’d use a similar construct:
s = '^hello ^world' re.sub( '\^(.)', lambda match: match.group( 1 ).upper(), s )
A caret means “start of the line” in a regular expression, so we need to quote it with a backslash.
Does it work? It does. The network is especially quick to learn
. ^ combo, representing the end of a sentence and an uppercase letter at the beginning of the next one.
The trick described above is meant for text. People have used char-RNNs for modelling other stuff. It is conceivable to use a similar gimmick for source code, or music, for example to insert bar markers - that might help a network to learn the rhytm.