Shout Box talking RPi: Trials and tribulations
September 9, 2013
A nice perk of LMR is the Shout Box, LMR's own little chat room. Many times I've been struck by motivation to solder/etch/hotglue/bodge something just through the good maker-mind energy in the SB but, alas, I would have to hang up the phone, if you will, if I were to head to the workbench. "Gee," I thought, "wouldn't it be awesome to have the Shout Box read to me so I can laugh and cry and work on a project all at once?" Yes, and it is awesome.
Even though I'm entirely entrenched in the world of Microsoft, when I got my Raspberry Pi I had a feeling that Linux would finally have a context that would make it applicable to my universe. A quick search with the Googles made me feel that even with my low-level understanding I could easily cobble together something that could make my RaspPi a Humble Narrator of the Shout Box.
Intitially it looked easy. From my perspective the SB magically appears in the right margin and updates when needed. I couldn't readily think of an easy way to filter out the SB content from the rest of the data. Fortunately there is a page that is SB only, all html:
Almost immediately I found wget which can download webpages from the command line. So I set about making a bash script that would feed the processed data to a TTS program. For the TTS I went with espeak. I wanted to use Google's translation TTS feature but the method of authentication and feeding it a file from bash was and is over my head. A plus for espeak is it takes input from a file or a stdout or you can just type in what you want it to say. Very flexible. So the plan was to download the SB page, isolate the shout line, save it to a file, extract the shouter's name with the colon, replace the colon with " says," save that to a file, extract the shout itself, save that to a file, and then feed the two files to espeak. Easy breezy, right?
The problem was when shouts contained non-UTF-8 characters like Ω or £ it would send the script into wonko-bonko mode. Who knew (from my little MS cave world) that these extended characters were incompatible with The Unix? The result was that it couldn't save shouts with those characters to a file for comparison and it put a hitch in the plans. That took a lot of time to work out. It messed up comparing the strings as well. I thought I was pretty savvy for trying that. Not so much.
Fortunately 6677 told me that TinHead had already been down this path and had success scraping shouts out with Python. A Google later and I found this node:
Upon seeing the page I remembered when Gareth was successfully shouting from afar with TinHead's script but I was mired in less substantial details back then and their achievements were soon forgotten. Thank you, 6677. Thank you, TinHead.
It was because of TinHeads pioneering that I learned of BeautifulSoup, a great tool for extracting information out of .html files. With that I was able to isolate the meaningful name/shout content from the tag-filled SB page. And british pounds and euro symbols were no longer sending the RPi into tailspins.
The output file was a list of "Shouter: Shout" with one per line, like you would see it in the actual SB. Each line was saved to its own file. I used the cut command to make a file out of the "Shouter:" and a file for the " Shout." They were then fed into espeak and they were vocalized. Eureka!
The problem though is that when the SB is hopping and the shouts are flowing fast you would be hard pressed to reload the info, read just the top shout, and not expect to miss a shout or two. Unacceptable. So I thought I would have the script read the first shout, archive it, pause a moment or two, reload a new shoutbox page, compare the archived shout file to the new shout files, find a match, and then read the shouts back from there, essentially from where it left off. The problem was after using the cut command to break the shouts up from the username the files were saved into some Nroff format that Unix likes to use. It adds sneaky formatting code for printing and such but it just shows up as different sized files. So when I would do a string or file comparison to detemine what was the oldest shout, they never matched up even if they looked like they should have.
An emacs vs vim note:
I tried both, but with emacs it immediatly informed me that the cut files were "Nroff" and the others were "fundamental." In the same amount of time I still couldn't figure out how to open a file in vim. Noob user says +1 for emacs regarding the seeming holy war between the two.
Super-uber-monster-mega-Skittles-rainbow-type-hyper-kudos to mogul for catching my frustration in the SB and recommending the troff command to convert them back to ASCII. It worked like a charm and all right when I was at my wit's end, seriously. From there I could remove all that Unix line formatting gibberish that was keeping files from matching up like they should.
After that it was a matter of sorting out some for-next and do-while syntax in this new language and the espeak module took it from there. It had a stuttering problem with longer files but piping the output to aplay eliminated that. I slowed down the speech rate a bit and use the familiar the colonial dialect of the United States. The default british voice was just not nasally enough for me to understand. Although espeak is not nearly as clear as the TTS from Google I find it more than acceptable. However at the default speed it sounded like a computer talking though a bag of marbles. Slower is better in this case, IMO.
Be warned , the code used is less than elegant but I tried to added meaniful comments for understandability. I thought bash could pull it off single handedly but I needed Python for the BeautifulSoup so you'll need both files.
I would have just gone pure Python but it didn't seem to come as naturally as Unix speak. With my patience wearing thin I decided to use both, so there is a Python script that gets run from the bash script. Keep it in the same directory as the main script if you want to toy with this. I'm sure there is a more eloquent way to go about it but I never claimed my programming was pretty. As my first RPi/Linux endeavor I was primarily interested in making it work rather than having it glisten in the binary light.
Many thanks to TinHead for paving the way ahead of me and Gareth for doing his testing. Thanks to 6677 for reminding me of their toils from many moons ago and to mogul for his sagey programmer's advice. Without any of the aforementioned this post would not have happened for some time, if ever. I mean, troff? I never would have suspected it to be the answer.
Shout Box talking RPi project page: