LingQ’s are essentially strings, so an ordered set of characters. So if you open a text, words shown in blue are those strings that don’t match any of those already existent in your list. I fairly doubt that the list contains doubles or stuff gets marked as blue although allready in your list. The respective datatypes and algorithms to work with strings and perform the tasks required here are part of the standard libraries of any high level programming language I have come across thus far (I’m not a programmer, though). So I fairly doubt the LingQ staff had to create this on their own.
If a language tends to contain lots of different forms of the same words (many cases, tenses, plural forms, grammatical gender, a high level of agglutination, …) you will come across blue marked forms of words you already know for months every know and then.