Can someone explain what algorithm or system is used to segment words in Japanese on LingQ?
What is used by default for imports and what for the “Japanese Word Split Optimizer”?
First it was said that machine learning was going to be used, specifically ChatGPT. And the popup says “Optimize with AI.”
But now I read “ichimoe” which points to https://ichi.moe/ which in turn appears to use GitHub - tshatrov/ichiran: Linguistic tools for texts in Japanese language
This doesn’t seem to be related to machine learning at all and use a more traditional lattice based tokenization like MeCab.
Thanks.
1 Like