Would it be possible to just stop trying to force AI to split Chinese words properly?
This is a solved problem. Here is a library that does the job quite well:
Here are some pretty basic sites that do it, also:
We don’t care whether it uses AI or not. We just want it to work.
Thank you.
Thanks for your feedback and suggestions. I’ll forward this to our team.
It seems that after the first disaster of doing that with Japanese, the same happened with Chinese, which was just fine for me before. The splitting was reasonably good and now it sucks horribly, producing an enormous number of agglutinated nonsense words, inflating the unknown new words. I have been avoiding to study Chinese and focusing on other languages meanwhile but one temporary solution that I saw with the guys studying Japanese, who faced the same problem, is to edit and regenerate the lesson. It is an extra step but it seems to better recognize the splitting.
It is great that you gave a proper example with a technical solution and I hope they can implement it soon.