Google colab notebook to select a language -> course -> lesson and then to run natural language processing from spaCy on the text

jamespratt · December 19, 2023, 9:00pm

Cool! My little experiment worked.

I have worked out how to drill down to select a lesson from amongst my courses on lingq and then to run that lesson through spaCy’s excellent natural language processing.

Here is my Google colab notebook.

And you can see the results here analysing an English text:

github.com

jamiepratt/lingq/blob/english/lingq.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOXrYn84DyqjEuGRmOlmdNd"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0228992868ef4147909348efc5811203": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DropdownModel",

This file has been truncated. show original

And a Polish one:

github.com

jamiepratt/lingq/blob/main/lingq.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOxUBVtKE+2QMyERltA4YK2"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0228992868ef4147909348efc5811203": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DropdownModel",

This file has been truncated. show original

I have not found another way in Polish to be able to correctly lemmatize text, ie. to find the base form of verbs, nouns and adjectives spaCy is also able to identify how verbs, nouns, adjectives, etc have been inflected - what gender they are or they are agreeing with, what case, whether they are in the singular or plural form.

Now that I have worked out how to run my lingq lessons through spaCy I will be able to chunk the text into sentences or noun phrases and be able to build up a db of sentences that I can then query so I can get all the sentences with an example of a male noun in the nominative case for example. All sentences with plural nominative nouns. The Slavic case system is notoriously difficult and I want to work out a way to mine my lingq lessons for examples of uses of the different cases in material I am familiar with so that I can do some extra work on understanding where cases are used and get familiar with the rather complex way that the endings of adjectives and nouns change depending on case.

jamespratt · December 20, 2023, 7:45am

I have split off the fetching of all text content from lingq into a separate script which anyone can use: