Best Way to Generate Subtitles?

Cinderela · September 28, 2022, 12:36pm

Thank you. Where can I find this software/app? Does it work in Farsi?

Cinderela · September 28, 2022, 12:37pm

Thank you!

alex1029 · September 29, 2022, 12:28am

i tried getting this to work on my mac to no avail not sure what i did wrong

Cinderela · September 29, 2022, 7:50am

Live Transcribe & Notification - The best app I have ever found.
Perfect for transcribing news or programs where the speech is clear and there is not much background noise. Also effective for movies but, of course, less so.

bamboozled · September 29, 2022, 11:12am

Thanks for the hint! I wasn’t aware of whisper, so I just tried it out.

The result is pretty good, not really worse than Amazon, but I only tested 1 minute of audio. Because this software is slow. One minute of audio took 20 minutes to process. This would result in something like 30h for a standard 90m podcast? Unfortunately this makes whisper unusable for me. For the record I ran this on a MacBook Pro 16 from 2019 (Intel) and used the sample code from the website:

Maybe someone else can try and share their numbers. I don’t know a thing about machine learning, so I probably did something wrong. If there are any performance tips, feel free to share them. Thanks in advance!

Also worth noting is that whisper doesn’t seem to be set up for long-form content: Stops working after long gap with no speech? · openai/whisper · Discussion #29 · GitHub
So it looks like I’ll still have to feed “Big Tech”, for now.

@Cinderela
The instructions are here: GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision
and Persian is supported, please feel free to share your results, I would be interested.

Bcpt · September 29, 2022, 7:02pm

Yes actually Whisper takes forever on CPU. However if you run on GPU the speed is faster. Using the small model I could process a 30 minute audio clip in approximately 12 minutes.

You can see the approximate times on their website I believe, but on a gpu the large, accurate model should approach real time while the fastest model should be around 32x faster.

For French I found some errors using the small model so I exclusively use the large model. I have been using it for tv episodes which are 21 minutes long. Each one takes approximately 18 minutes to process, and I just simply queue 10 or more of them up before I go to sleep.

If you don’t have a strong GPU, you may consider using google colab. It’s a free resource using googles hardware. I’m not too clear on the limitations but for whisper at least I tested it and didn’t run into any over the course of about an hour. I imagine it would work for you for a 90 min podcast once in awhile. Make sure you switch the runtime environment to gpu.

As for long form content I did see some discussion about it but I hadn’t personally encountered any problems with it so I don’t know much about that.

The only thing that whisper annoys me a bit about is the time stamping for the generated subtitles can be somewhat off and needing manual adjustment. Of course as noted on their website, some languages perform wildly different than others.

rickvan_wichen · September 30, 2022, 2:12pm

OMG bro! I’ve looking for something likes this for months! I’m so happy. I’m learning Bulgarian and finding transcriptions is like the hardest thing ever. Thank you!

rickvan_wichen · September 30, 2022, 2:13pm

OMG bro! I’ve been looking for something likes this for months! I’m so happy. I’m learning Bulgarian and finding transcriptions is like the hardest thing ever. Thank you!

pixla · September 30, 2022, 7:56pm

How to make text from audio you need blackhole and Microsoft word. I hope you now share more your sources (text and audio) on lingQ with this tools Simple automatic transcriptions using Microsoft Word and the Blackhole audio driver. - YouTube

bamboozled · October 19, 2022, 4:21pm

I don’t know if anyone else has been looking for a more performant alternative to the original Python implementation, but I found this gem:

It’s a reimplementation in C/C++, optimized for CPU using SIMD , threads etc. Overall I’m very impressed with the results, occasionally it fails or gets hung up on one sentence, which then repeats endlessly. But this only happened on 2 out of 23 files I tried.
The accuracy is actually quite impressive. For Chinese this is on par with Amazon I’d say. I have also tried Romanian which probably hasn’t had much training data, for example Google Cloud struggles quite a bit; Whisper gets better results, although it is not close to the accuracy it achieves in Chinese. But I’m not looking for perfection anyways. Here are some non-scientific performance numbers, all tests were done with the medium model on a MacMini M1 using 4 threads (adding more doesn’t improve the results) as of b4a3875.
Input length in seconds / time to output in seconds
1539 533
2389 697
1755 587
1810 531
1731 553
1733 540
1491 494
1976 659
2181 651
2282 733
Total:
18887 5978
So, roughly 3x realtime, 3 hours of input take about 1 hour to process. The above results were with Chinese, but I couldn’t see a difference with Romanian.
The software also supports vtt and srt subtitle output. Please note that LingQ will ignore the first subtitle line after importing, I already reported this bug. Just use txt for the time being.

rafarafa · October 19, 2022, 6:23pm

I didn’t try the python implementation but I decided to give this a try and it seems to work well for japanese. Granted I only tested it in a couple files, and the speaker was speaking very clearly so take as it is.
What model did you try? I went for the base since it’s what they suggested in the repo but I’m not sure how it fares with respect to quality. As I understand it with the bigger ones (medium, large) you trade computation for more accurate subtitles?
Thanks for the share btw, very interesting.

bamboozled · October 19, 2022, 6:52pm

My tests were with the medium model. Maybe I’ll experiment with other sizes later, but currently I’m satisfied with accuracy and speed, especially for Chinese. Romanian is already a bit flaky, not sure I could go lower.
The original paper (https://cdn.openai.com/papers/whisper.pdf) includes “word error rate” (WER) data, starting on page 23, this can serve as indicator for the accuracy across the various languages. This might help to determine which model to choose.
Interestingly Japanese seems to achieve significantly better results than Chinese, even though the latter had quite a bit more training data (see Figure 3 on page 7). Curious.

Btw. one big thing I forgot to mention is that Whisper supports punctuation, this is not a given even for payed services.

alex1029 · October 20, 2022, 2:18am

guys im so bad at this how do you even begin to use the program !

bamboozled · October 20, 2022, 6:32am

Well, I only tried it on macOS, on Linux it’s probably even easier. I don’t really know, but compiling on Windows is probably a world of pain.
[Edit: actually you can just download a Windows executable from the automated build system on Github - look under Actions → Artifacts]
Please note, all commands are provided in good conscience, but I assume no liability if you screw up your system

Prerequisites:

C/C++ compiler
git (version control, preinstalled)
FFmpeg (for converting the audio files)
macOS:
On macOS: if you have Xcode installed, you are good, else open the terminal and run:
xcode-select --install

You can check your compiler:
clang -v

If you have https://brew.sh/ installed type to install FFmpeg:
brew install ffmpeg

(MacPorts should work as well)

Linux:
Potentially everything is in place already, on Ubuntu you may need:
sudo apt install g++
and

Cloning the repo and compiling:

Pre-processing the input file:
(change input.mp3 to your input file):

Model download:
For example. the medium model:
bash ./download-ggml-model.sh medium

Example command:
This is the command I used for my tests thus far (medium model, Chinese language, srt subtitle file output):
Choose the language:

github.com

ggerganov/whisper.cpp/blob/78f166174f126345ed87cc8f6941af1905c4a0f2/whisper.cpp#L54


      
          // available whisper models
          enum e_model {
              MODEL_UNKNOWN,
              MODEL_TINY,
              MODEL_BASE,
              MODEL_SMALL,
              MODEL_MEDIUM,
              MODEL_LARGE,
          };
          
          
static const std::map<std::string, std::pair<int, std::string>> g_lang = {
              { "en",  { 0,  "english",         } },
              { "zh",  { 1,  "chinese",         } },
              { "de",  { 2,  "german",          } },
              { "es",  { 3,  "spanish",         } },
              { "ru",  { 4,  "russian",         } },
              { "ko",  { 5,  "korean",          } },
              { "fr",  { 6,  "french",          } },
              { "ja",  { 7,  "japanese",        } },
              { "pt",  { 8,  "portuguese",      } },
              { "tr",  { 9,  "turkish",         } },

Output format:

Beam Search:

Initial Prompt
The --initial_prompt option gives the model a hint what came before the first audio sample, it can help with special vocabulary or difficult to spell words here is an example:

Other options:
The -pc option gives colored output, according to how confident the model is
-pp will print occasional progress messages, e.g. 5% done

Batch:
You can also incorporate batch processing, here are some ideas:
for f in .mp3; do ffmpeg -i “${f}” -ar 16000 -ac 1 -c:a pcm_s16le "${f%%.}.wav"; done
for f in *.wav; do ./main -m models/ggml-medium.bin -l zh -pc -pp -osrt -f “${f}”; done

alex1029 · October 21, 2022, 4:27pm

thanks im gonna try it!!

michelleschilz · October 26, 2022, 7:31pm

I was trying to sign up for this service, but couldnm’t get past the initial verification step. Not getting a text on my US phone number, and they don’t offer an email alternative.

bamboozled · October 26, 2022, 8:00pm

You mean iflyrec? I think they only accept Chinese phone numbers, and even then the real problem will be the payment. I don’t have an account there but I doubt that their service is somehow better than “Big Tech”. It’s probably not worth the hassle. Personally, I have moved on to Whisper and create transcripts myself. But I always found Amazon’s service to be adequate and reasonably priced as well. I you are interested I could send you a sample transcription privately. They only require a credit card IIRC.

alex1029 · October 26, 2022, 9:49pm

heyyy how much did you end up spending on amazon? and how is the process?

bamboozled · October 27, 2022, 4:24am

I think I spend about 100€ on transcripts. In the free tier you can get 60 minutes for free each month for one year, if you exceed that you pay. The pricing depends on currency and region, see here: Amazon Transcribe Pricing – Amazon Web Services (AWS)
The process is quite simple, create a storage bucket on S3 and upload your audio files, then add a transcription job. Everything can be done using the website, but is available via API as well. As for options, I found the “speaker identification” to help with accuracy, just enter the number of speakers. Also don’t forget to the select the subtitle formats. Also, I’ve had the language selection reset on me, so when creating multiple jobs, make sure to select the language for each item.
Transcribing with the AWS Management Console - Amazon Transcribe
Getting Started with Amazon Transcribe