Japanese learning app prototypes

Challenge

I’ve been learning Japanese since 1998 and while I’m pretty good for a gaijin living away from Japan, progress has been slow and sporadic. I feel that part of the problem is that Japanese learning apps and web sites aren’t interactive and smart enough. I’ve tried dozens, some of them for months, and they each have many severe usability issues or glaring mistakes that confuse and discourage learners. For instance, here are two examples of mispronounciations from the iOS app Nihongo, a modern dictionary:

The first example (1) should be pronounced いれて (irete) in this context. That kanji is shared by multiple verbs, so it’s sometimes pronounced はいる (hairu) depending on surrounding words and context. No learning app performs this analysis right now. It’s even more complicated with proper nouns (example 3). These often have dozens of exceptional pronounciations, and even Japanese adults find it challenging to pronounce the names of small cities or unusual last names. With deep learning, this should be solveable given enough examples, e.g. using a RNN like LSTM.

During my spare time, from Q4-2017 to Q1-2018, I decided to attempt to explore the feasibility to single-handedly create a cross-platform app that would help me and my kids practice Japanese and learn more efficiently.

I was also hoping to:

• Get practical experience with modern Javascript, portable mobile app development (Android and iOS).
• Eventually, integrate deep learning, e.g. recognizing handwritten kanjis, automatically inferring the pronounciation of ambiguous words, etc.
• Assess if sustainable revenue (e.g. $50K) could be achieved within a reasonable timeframe (e.g. 1 year)

Non-goals:

• I had zero interest in learning both Android and iOS native app development, e.g. using Java/Kotlin and Swift/Objective C. That’s too many programming stacks for my taste.
• I didn’t want to have to buy a Mac - I’m happy with Windows and Linux, I don’t have the patience to master a 3rd OS.

Main technical concerns

• Performance - e.g. how unresponsive is the app?
• Ease of debugging, automated testing, etc. I wanted this to be a fun learning experience using modern tools, e.g. avoid hair-pulling debugging sessions using printfs and long test cycles.
• Robustness - even if only 5% of users experienced crashes, there was no point in attempting commercialization.

Two major options

I explored popular frameworks to develop portable mobile apps. I quickly eliminated:

• Unity. I knew Unity already but I felt its was overkill for a “small” app. I quickly eliminated this option because the starting time was too high (more than 1 second). I was also concerned that integrating hardware-accelerated deep learning would be tricky down the line.
• Xamarin. I tried a few apps and was unimpressed with the runtime performance. C# again presented potential difficulty down the line to integrate deep learning.
• Most Javascript-based frameworks had major issues, e.g. limited developer adoption, poor performance, etc.

The two options I explored further were:

• Qt, QML and C++
• React Native

1st attempt: using Qt, QML and C++

I first tried developing the app using Qt, QML and C++, since I had recent experience with this development environment, albeit on Windows Desktop. A few lessons I learnt while developing small proof-of-concepts:

• Getting started with Qt + Android was relatively easy. There are good samples and very little C++ is required.
• A Mac development environment is required for iOS. I was able to install a Hackintosh VM over Linux. It was a bit slow but it worked. There was however the concern that future Apple updates would break my development environment. So I was mentally prepared to buy a real Mac if I decided to go ahead with commercialization.
• I needed to bundle a real Japanese TrueType fonts to avoid the (Han unification nightmare)[https://en.wikipedia.org/wiki/Han_unification]. That took several MBs, slowing down incremental compilation and deployment. The solution was to bundle this in a separate resource file than the QML files. That way frequent changes to the (relatively small) QML files didn’t require re-uploading everything.
• After optimization, turn-around time to recompile and redeploy on Android was about 15 seconds (acceptable), and about 1 minute on iOS. (painful).
• Qt offers many options for displaying UI. I used the relatively new Qt Quick 2 and was pleased with it. The runtime performance was excellent even during stress tests - a consistent 60fps when displaying a few screenfuls of sentences and <100 ms when loading new screens.

  

• Developing small components and simple logic with QML (based on Javascript) was relatively simple after re-reading some of the documentation. But as my prototypes grew more complex, the automatic evaluation of QML dependencies got in the way. Building a large application (like our engineers at Fortem built - Omnipresence 3D) using QML still seem to require frequent unintuitive software architectures or a mix of C++ and QML that don’t feel immediately elegant. I guess with a few more months of practice it could have become second nature. But there were other issues…
• I was originally developing on Windows 10 and often ran into the 260-character path limitation in some of the Android build tools. This was very annoying - the error messages were often cryptic. I tried Windows junctions (equivalent of symbolic links) and while that solved some problems, it wasn’t a simple long-term solution with git. I ended up simply keeping my paths really short, e.g. renaming “claforte/learning/tutorials/001_some_kind_of_tutorial” into “cl/lrn/tuts/001_smof”. Felt like MS-DOS.
  
• While Qt comes with useful small components, the ecosystem is limited. Building an eye-pleasing interactive design would have likely required me to spend weeks doing graphic design work for which I have no talent nor interest. That’s the #1 reason why I decided to try…

2nd attempt: using React Native

I built my first prototypes with Create React Native. The first impression was very, very good - building common features is really easy. The documentation is complete enough. The tools are fast. The debugging isn’t great - I wish it was better integrated with vscode. Javascript has evolved into an elegant and powerful language. I quickly ran into problems though:

• Node packages version conflicts: I bought a couple of recent books (less than one year old) to read on best practices and do some coding exercises. I could never get any of the examples to compile the first time around. There were tons of deprecated APIs, changed interfaces, etc. Some of it I suspect was due to npm’s inability to keep track of actually installed versions in code control - i.e. solved by yarn. In general, the whole Javascript ecosystem feels like Dubai. Like the aptly-named Dubai Torch Tower, it looks amazing and can be built in record time… but it takes only one cigarette to set whole building on fire. Except that this isn’t an isolated incident - there are so many interdependent packages that this happened every couple of weeks.

• Dealing with native packages was a lot of hassle, and again, required the Apple and Android toolchain.
• Still ran into 260 characters path problems on Windows. Same work-around - keeps paths you control really short.

3rd attempt: using Expo

I then upgraded to Expo and boy, am I happy I did:

• Expo curates and integrates the most common, high-quality React Native packages. The native packages are precompiled into standard apps that streamline development. This eliminates most incompatibility problems I ran into previously.
• No more need to compile and deploy on Android or Mac. I never had to deal with adb again. All you do is install the standard Expo app on your phone, scan your QR code, and you’re done.

Developing with React Native and Expo was a load of fun:

• Hot-reloading works great all the time. It often took less than 5 seconds for my code changes to be reloaded.
• Runtime performance isn’t as great as Qt but still very good. I got 30fps without difficulty. I was however careful not to load too much stuff in each screen, e.g. loading only the first 10 matches in the dictionary at first.
• The ecosystem is exciting - lots of useful packages, and most of them don’t need native compilation. With expo’s mindshare increasing, these components are generally compatible right away. Some of these components are beautiful and helped me achieve a polished look easily.
• Automated testing with Jest was really fun and fast. The only issue I ran into here was debugging the client-server communication. I know I should have written more mocks, but for my purposes, it felt simpler to simply have the client tests cover some of the integration tests as well.

(BTW, my daughter Emily stars in the videos, and my wife Misako recorded the audio sentences.)

Other technical issues

Afterwards, the biggest and repeatedly annoying issue I ran into was Javascript language incompatibilities. The node ecosystem uses a tool called Babel to “transpile” Javascript extensions (including yet unofficial advanced features that may or may never be approved) into whatever version runs natively on the platform, e.g. Android. I bumped into obscure bugs because Babel was either incorrectly configured (a real pain in the ass depending on packages) and in at least one case, because Android’s javascript engine itself was buggy. (“for of” loop IIRC)

Asynchronous code is also both a blessing and curse in node. IMHO there are too many ways to express asynchronous concepts and most tools (e.g. debuggers) can’t handle it – especially when Babel transpiling makes things even more complicated. Tracing small asynchronous function was ridiculously complicated. At least, with the momentum behind React Native and Expo, I’m hopeful these problems will get resolved in the next year or so.

Server and NoSQL database

I used feathersjs and the default NeDB NoSQL database to build a quick-and-dirty server including a real-time SocketDB API. The Feathers CLI generator are very helpful. The documentation was so-so. (It’s since improved a bit.)

One task that took much longer than I anticipated was importing and processing the Japanese dictionary. It was provided in a 90MB XML format, covering ~170K entries and definitions. I first tried to load the whole thing at once, and ran out of memory (12GB). I had to implement a sax parser. It took me a while to get the abstraction at the right level so that code wouldn’t be tag-specific all over the place. Another issue is that the loading code could run really fast (thousands of entries per second) but uploading them to the server to be saved was 20 times slower. The sax parsing code was based on callbacks, while the Feathers client dealt with asynchronous functions I had to await for. My first attempts at plugging these two together always resulted in promises or connection timeouts. It took me a day to step back and realize the simplest way was to implement a two-pass process instead.

For the longest time (2-3 days) I had what looked like a performance problem processing relatively simple NeDB queries. As I ran more tests, I realized I had bumped into another Babel+Android bug that was causing some of my asynchronous functions calls to abort early. I ended up rewriting much of the code to simply avoid newer Javascript keywords like await/async, for of, import, etc.

Conclusion

Overall, this learning experience has been positive. I did however put this specific Japanese app project on the backburner. Mainly, I don’t think I can turn this into a profitable product - the market for Japanese learning software is just too small and the effort to turn this prototype into a fleshed out product wouldn’t be worth it.

For the time being I’m focused on other exciting projects that involve more Deep Learning and 3D graphics.