Importing Audible transliteration full of errors

I successfully imported Maurice Druon’s ‘Le Roi de Fer’ from Audible, but I found the transliteration extremely buggy. You need to train your AI on the text as well as the audio recordings. There are transliteration errors in practically every paragraph. I ended up purchasing a hard copy of the book in French because so much of the text as transcribed by Lingq made no sense at all. In a way it’s been good for language learning because it’s forced me to dig deeper into each sentence, which is rewarding because I am a total nerd, but anyone expecting a free ride from Audible to Lingq will probably be doomed to disappointment.
Here’s one example: “Que comptiez-vous donc?” should have been “Que contiez-vous donc?” The word “comptiez” should have been “contiez”. Lingq translated the first sentence as “What are you counting?” which makes absolutely no sense at all. If the transliteration of that one word had been correct, the translation of “So what are you saying” would be right. That is only the most recent, and I’m only on chapter 2. Login - LingQ

Once you retrain your AI please let me know. I’ve enjoyed the deep dive into advanced French by carefully proofreading the nonsensical transliterations, but it’s a hard road when you have to second guess everything and really want to just dive in and enjoy the story.

This is related to the whisper transcription feature, not LingQ’s transliteration feature for pronouncing words, right?

I’ve worried about this myself with Korean.

Mostly because the phonetics of the Korean language and how some words/consonants/vowels can be mistaken for others, even by native speakers (for some words).

It’s not unrelated. If Lingq is not getting the correct words in English, then pronouncing the wrong words correctly doesn’t help. You need to proof Lingq against the original text and check the spelling of words that make no sense in context. In French/English the letters n and m can sound alike when read, but words containing them mean totally different things. Sometimes it comes up with words that don’t exist at all, such as “eusure” for “eurent” (had, according to Google translate), which it just ignores. To find out what the word is supposed to be you need to look at the original text. With Audible, that’s what the voice actor is reading from. If you don’t have the original text, then using Lingq with Audible is likely going to result in frustration.

Unfortunately, the best we have is Whisper and neither Whisper, nor LingQ can compare the generated text with the original text. If we had the original text, then we wouldn’t need Whisper’s generated text.

Whisper, over time should get better at determining the correct words from the context. Until then, This is the best we can expect.

2 Likes

Ummm, you can get the original text. I already had the Kindle English translation, so I knew how the story was supposed to go. I was getting ChatGPT’s help originally and ChatGPT alerted me to the misspellings, so I went to Abebooks.com and picked up the original for under $10 with free shipping, so yes you can get the original text the voice actor is reading from. If Whisper is purporting to accurately convert Audible books to lessons, then it needs to be trained on the text, not just what it thinks the voice actor might be saying. One needs to listen with the hard copy book in hand, and not look at Lingq’s synchronized view. I’m having fun with this deep dive because I’m familiar with the material. I’m learning a lot by proofreading it line by line, digging into the French language to stroll the streets of 14th century Paris, but I can see how it would be confounding for someone who had not already read an English translation of the work and didn’t realize that Whisper was actually producing gibberish.

1 Like

True, but LingQ has a feature where you can turn on “Transliteration” (also known as Romanization for some transliterations of language scripts — like Korean, Chinese, and Japanese into something English speakers can pronounce — into latin characters) to help users pronounce words.

Those transliterations appear on a different line to the original text.

The feature that you mention is LingQ’s Whisper Automatic Transcription feature, which happens to process a spoken word differently then the one actually used since the model needs improvement.

This can result in “near sounding worse” or completely incorrect words being produced by Whisper that can “mess up” your pronunciation of similar words if you study with the flawed script produced by Whisper in that instance.

I highly recommend any user using Whisper obtain an official transcript or text-based copy of the materials they put through the Whisper transcription feature especially if they’re a beginner in that language.

Advanced users might be able to more easily draw on their vocabulary to correct errors in transcripts produced by LingQ’s Whisper feature. Still, as mchipp points out, this is likely to be a frustrating roadblock to beginners if they overlook the mistakes produced by the Whisper transcription.

And some materials (as Mycroft points out) don’t have any written source material available, so in those cases, it is “the best we have.” Still, beginners should use that Whisper-generated material, knowing it could be full of mistakes.

Whisper is definitely far from a “plug-and-play” type feature.

P.S. nfera provides a useful chart on this feature here:

Drawing on your vocabulary is useless if the words produced are completely wrong. Lingq should put out a “caveat emptor” message informing users that the Whisper output cannot be trusted. It can be a good learning tool for those willing to take the time to excavate the text, sentence by sentence, with the original language text plus ChatGPT and/or a good dictionary. ChatGPT is great because even when words are correct, it still may make no sense in the context of the literary work being translated. For example consider this phrase in Le Roi de Fer describing the wares laid out for sale in the merchant’s hall: “Broderies, dentelles, soirées, velours et camelins…” Lingq translated it as Embroidery, lace, eveningwear, velvet and camelins" because it mistook the word soieries (silks) for soirées, and left camelins (camels) as camelins. Eveningwear is plausible, but camels, not so much. Just picturing camels for sale in a medieval merchants hall made me laugh. I asked ChatGPT, which is quite familiar with Maurice Druon’s Roi de Fer, and it informed me that “Camelins” actually refers to a type of fine cloth that was made partially from camel hair and partially from wool. It was a well-regarded and somewhat expensive fabric in medieval Europe and was used for various purposes, such as making garments."

So this is fun for me, in the same way some people like solving crossword puzzles, but as a standalone language learning tool it’s not ready for prime time. Not even close. I’m making corrections in Lingq, and sharing the chapters I’ve completed, but at this rate it’s probably going to take years to finish accurately translating just the first the book in the series into usable Lingq lessons. People really should be warned before they invest the time so they don’t get frustrated. I can see how this would get old very fast, even for advanced users.

1 Like

It should be understood that asking AI to automatically transcribe audio is not going to be perfect. In fact, even human transcription is mostly imperfect. This feature, using Whisper AI, is provided to enable users to make use of audio material when they don’t have the original text. Of course, you wouldn’t use Whisper to generate an automatic transcript if you already possess the actual text. Whisper will never be as accurate as the original text. Its accuracy depends on language and the clarity of the speaker or speakers in the audio. It needs to be taken for what it is. A great way to get some text that in many languages is a very good representation of the spoken audio.
As for the odd mistake, really this shouldn’t be too much of a concern. The feature is really not for beginners who should be using beginner content from the library. This is for users interested in accessing long form content like YouTube videos and podcasts. Intermediate and Advanced learners should better be able to judge the quality of the transcripts. And, better able to ignore or gloss over the odd inconsistency.
It really doesn’t matter that much if the odd non-word or incorrect word is shown. In the big picture, each word’s impact is negligible on a word by word basis. It’s the overall effect of seeing tens of thousands of words in multiple contexts that will grow your ability in the language. The odd strange word will just get ingnored by your brain over time.
Of course, perfect transcripts are nicer and better to have, and, you can fix them as you are doing. But, also, just fine to go through, save the words you are interested in, even incorrect ones and not worry about it too much. It’s the overall that matters. The words that matter will appear in multiple contexts, most of which will be correct and your brain will gradually come to know what they mean. Steve is the first to say not to expect perfection from your self in language learning and the same applies to the materials you are learning from. Your brain is designed to sift through the cruft and find the patterns that matter.

8 Likes

I absolutely agree with OP that working with a transcript full of errors in the target language should be avoided at all costs.

That said, I am not sure that using the Whisper function here was the best way to go. It would be preferable to buy the book on Google Play (it is available, l have checked), remove the DRM protection using Calibre or Epubor, and then import the text into Lingq either automatically or manually, which is my preferred way of doing it as I like to have control over how my chapters are divided.

This way you can be 100% certain that the text you import will be fully accurate, saving you time and frustration.

The Whisper function is to be used mainly to transcribe podcasts found on YouTube, but for which no transcript was made available. It is far from perfect, and will have a much harder time recognising rare French words.

If your primary intention is to read the novel in French after having listened to the audiobook, I highly recommend you follow my suggestion.

4 Likes

It’s not their AI. It is Whisper–Open AI’s AI. LingQ can’t train it. You are also free to use Whisper outside of LingQ with a higher quality setting. Maybe that would work better for you?

Whisper doesn’t purport to accurately convert Audible books to lessons. Whisper does not know anything about LingQ. LingQ is using Whisper as a tool to help those who may not have a transcript for some audio content. If you have access to the real book or real transcript that is always better. Feel free to find another transcription tool if you think you can find a better one. Or if you know of a better one, then help us all out and point us to it.

If the book is copyrighted you shouldn’t be sharing the lessons anyway.

I’ve had the opposite experience. I’ve found it extremely useful to get transcripts for many things that I don’t have a transcript for. It’s not perfect but it’s been nearly so for me (in my experience, ymmv). The fact is, as far as I know, there is not a better tool out there. It’s free, and pretty darn good. It seems as good or better than any of the paid for transcription services (automated…not manual) that I’ve tried out.

Lots of folks have asked if they could add it. Many of us were using Whisper directly, or indirectly and uploading our lessons to LingQ. Having it incorporated into LingQ has made it a much simpler process. I thought it was great they were able to incorporate it so quickly. Since many folks have been using it prior, I’m quite certain most consider it more than adequate. I suppose they could put a warning, but it’s quite easy to see when it is or isn’t working well for a particular language or a particular source and most of us just move on and not get upset at LingQ for it (it’s not their technology so what use is it to complain to them…go complain to Open AI).

2 Likes

Thanks for your suggestion, but my primary intention is to listen to the Audible book in French while reading the text at the same time. I like to listen to the entire chapter while reading the synchronized text in Lingq, and then go through the chapter in sentence view to fully learn what was said. That is a really bad idea with the current version of Whisper. The workaround in my case was purchasing a cheap paperback in French where I can see what the voice actor is reading in real time. I can go back later and mark up the missed words on that page. If I just ignore the synchronized text while listening, then my brain will absorb the correct words as intended. I won’t incur the cognitive load of having taken in the wrong word or phrase at the outset and then have to correct it. I’m far from an advanced learner in French–at best a stage one Intermediate or advanced beginner, but I find the podcasts/youtube lessons geared to that level to be boring and tedious. I really, truly don’t care about the cities in France or common mistakes people make when speaking as I will never see France again in this lifetime. I already have fairly decent comprehension of written French, it’s the audio that flummoxes me. Listening to a talented professional voice actor read a work of literature is like attending a great performance inside my head–I can visualize the scene as it was intended, in its original language. Then piecing together the meanings of the words so beautifully spoken bestows an additional pleasure, akin to solving a puzzle. I can go back and listen to the audio again with book in hand, which reinforces the correct textual meaning, not the bogus one Whisper provided to Lingq.

It seems that the problem is that you don’t understand about what you are talking about. I mean the transcription technology behind the seen.
Others have told you already but you keep not understanding.

At the moment, in the PLANET EARTH, it seems there is no better alternative to Whisper by OpenAI.
LingQ is using the best option that we have to create those kind of transcriptions, in multiple languages at the same time, from any type of audio source.

If you are able to invent a technology better than Whisper, we will all use it and you will be billionaire.

It is a big convenience and the alternative is to upload your ebook text along with the audio, or pay a native person to transcribe it for you (if you are rich).

It is your material, you buy the Audible, you buy the ebook if you want to, and you upload both privately.

Whisper is very useful for convenience also on podcasts, youtube videos, as others have already told you.

Personally, I accept that there are sometimes errors compared to probably 97% accuracy. It is probably not so good for beginners though.

Here is a short article about Whisper by OpenAI: https://www.xda-developers.com/openai-whisper/

Here is another short article about Whisper from OpenAI: Introducing Whisper

3 Likes

You are NOT using the Whisper transcription function for the purpose that LingQ incorporated it. It should be used (as others have said), for podcasts or videos that do NOT have transcripts available. While not perfect, it’s better than the options we previously had.

While I have tried it as an experiment, using an audiobook for Whisper, makes for a very painful transcript. Whisper has no idea who is speaking in dialog, or which sentences ARE dialog for that matter. Also, it has no idea which sentences belong in a paragraph. It will take some years before Ai is able to accomplish these things.

As others mentioned, it would be best for you to purchase the ebook, decrypt it with Epubor, and import the text into your LingQ course. If an ebook is unavailable, the best you could do, if you must use that particular book, would be to scan the book using the app “vFlat Scan” on your phone and import the text.

Good luck.

1 Like

Good point Mycroft. I am not using Lingq for its intended purpose, which seems to be devoted to the metric of amassing lingqs, coins, or learning streaks, which I have always had zero interest in. I wanted to learn French to connect more fully with the great works of French literature that I had only read in translation. Your advice is spot on for those whose goals are to amass words, points, or whatever makes them feel like they have accomplished something. Nothing wrong with that. Lessons and podcasts are a great way to get up to speed, but for me, the content itself isn’t interesting enough to be worth listening to multiple times. It all depends on what your goals are.

“If you point your cart north when you want to go south, how will you arrive?” --Ryōkan

Although I may not be in the majority, I’m sure there are other learners out there who use lingq because they love the language, the literature, the culture, and want to learn the language for personal enrichment, to appreciate the art of great writing as it was intended to be read, in the original. Only a fool goes to a ballet to count the steps, or views a work of art to calculate the cost of the paint. For myself, amassing words is a meaningless exercise, and I can’t force my brain to do it. Although one can get the bare bones of a good story in translation, the deeper art of the thing lies in the original language of its creator.

I originally wanted to start with Balzac (I read all of his works in translation) but Maurice Druon proved easier to acquire in Kindle translations. The French is also more accessible, having been written in the 1950’s, not the 1830’s.

I was overjoyed to find that I could import the Audible version of Le Roi de Fer as read by the amazing Jérémie Covillault, who is a revelation. He has a marvelous voice, and imbues every sentence with meaning, acting out the different characters. His reading provides inflection and nuance to every sentence that would be impossible to infer from the printed page alone, much less from a translation. They aren’t just words to be clocked, they evoke not just the environs of Paris under Philippe le Bel, but also portray the inner thoughts and dialogues of historical characters. That part is imagined, but the facts of the story are historically accurate. We will never know what Jacques de Molay thought in his cell as he awaited being burned at the stake, or what went through Philippe le Bel’s mind, but Maurice Druon imagined it. He sets out what he’s going to do in the prologue, and then jumps right in:

Mais l’idée nationale logeait dans la tête de ce prince calme et cruel pour qui la raison d’Etat dominait toutes les autres. Sous son règne, la France était grande et les Français malheureux.<<

Whisper is not inimical to appreciating literature, its inadequacies actually force one into a deeper appreciation of the text. Last night I came across this amusing mistranslation. Philippe le Bel surprises his adulterous daughters-in-law talking with one of the lovers in the marketplace, and Mes filles, mes filles came out “méfiez, méfiez”. He then asks “Qui est-ce d’un oiseau?” which was translated as “Who’s that of a bird?”. d’un oiseau should have been ce damoiseau, but I would have never dug into the meaning of damoiseau (meaning a young man who is not yet a chevalier, or knight). In modern French, Philippe le Bel would probably say “Qui est ce mec?” :slight_smile:

The brain learns from story, from what it finds interesting. Autodidacts are different than most people, and what appeals to us is finding our own way to the center of the meaning of things, not skirting the edges, passing a test, or amassing coins or words. I’m sure I’m not the only one.

I would highly recommend that such people do use the Whisper technology with an Audible work that captivates them, as long as they have a textual version of the original. It is a wonderful way to learn. Whisper’s mistakes hook you into remembering the actual word when you have to do your own research to dig out the true meaning. It’s quite different than having something handed to you on a plate in a literal translation. I’ll never forget the mnemonic of d’oiseau for damoiseau.

By the way, you are dead wrong about it taking years before AI is able to accomplish these things. It’s already happening, just do an Internet search. Whisper and ChatGPT (another indispensable tool for learning) are both from openai, and openai isn’t the only game in town.

You are correct that it will not be long, in the grand scheme of things, for AI to transcript (and translate) better than humans. It has some advantages. After all, it will be able to “listen ahead” and know immediately that someone is speaking a particular sentence and who it is and it will be able to use large language models to arrange the sentences into proper paragraphs. However, even though cars can drive themselves, that day has not arrived yet.

But as long as the original text is available for import into LingQ, it isn’t really necessary or desirable to transcribe the audio.

Personally, I don’t use the included LingQ material. I import all my own books to read on LingQ. It’s a great way to keep track of words and phrases I don’t know and whether they have come up previously (meaning I should make more of an effort to learn them).

Reading on Kindle is a great experience, since you can just touch a word or phrase and get the instant translation. But it won’t track them. So, while it would make a great list of words and phrases, you could do the same thing by just writing them down and the list gets to be quite long. LingQ lets you know the words and phrases that keep popping up. For me, that’s it’s best feature.

3 Likes