They aren’t interchangeable. Either use “that” or omit it. As has been thoroughly explained above, using “when” would change the meaning of the sentence.
Buuut, you seem to want to speak the way you like rather than listen to native speakers’ advice… so, go right ahead! 
Any resource that tells you “that” and “when” are interchangeable is wrong and should not be trusted. While a few instances of interchangeability exist, as a general principle it is flat out untrue.
Am I right in thinking that in some cases “that” or non-existence of “that” can mean “when” as a relative adverb in the context, but that they can never be changed into “when”?
“The day that she left Tulsa”
“She never came out and told me that I was wrong /
But all of sudden the light came on the day that she left Tulsa”
http://www.metrolyrics.com/the-day-that-she-left-tulsa-lyrics-wade-hayes.html
You will be less confused if you think of “that” and “when” as separate words, with separate meanings, and quit trying to swap one for the other. If “when” is meant, “when” should be used.
FWIW, I had never in my life heard the term “relative adverb” before this thread, and it does not appear in my English grammar reference. While the expression may very well be used by linguists and grammarians, and may be helpful in mapping English to other languages, it is not part of the vocabulary that non-specialist native speakers use when talking about their language. Moreover, in most of the examples in this thread, both “that” and “when” are functioning as conjunctions, not adverbs.
An adverb modifies an adjective, a verb, or another adverb. A conjunction defines the relationship between two clauses. “That” and “when” are different because they imply different relationships between the two clauses that they connect.
Incidentally, lyrics are generally not good grammatical references. Poetry (in any language) is more concerned with sound and rhythm than grammatical correctness. For example, in the lyrics in your most recent post, all three instances of “that” could be omitted. In fact, maybe that’s the rule of thumb you’re looking for: if “that” and “when” can be used interchangeably, you can omit them both. Counter examples welcome.
@kewms
"If ‘when’ is meant, ‘when’ should be used. "
The difficulty lies here. If “that” is used as a relative pronoun, I don’t feel any difficulty. However, it’s difficult for me to decide when to use “when” and when to use “that,” as a relative adverb in order to begin a restrictive relative clause, of course, without a comma. (Thank you for your replies.)
Edited. Inserted “restrictive” before “relative.”
In the original sentence of this thread, changing “that” to “when” and not adding a comma doesn’t change the function of “when”. Whatever you mean by “when” becoming a “relative adverb” if you don’t use a comma is a strange idea. If you mean it suddenly takes the function of “that” in this sentence because you left out the comma, it’s just wrong. It would still have a different meaning than originally intended by using “that”, but it would be written incorrectly. In this case, they are not interchangeable.
@LFJ
"Whatever you mean by ‘when’ becoming a ‘relative adverb’ if you don’t use a comma is a strange idea. "
It goes without saying that If you begin a relative clause with ‘when,’ this ‘when’ is a relative adverb. If you put a comma before ‘when,’ the clause is a non-restrictictive relative clause. Restrictive or non-restrictive depends on whether there is a comma or not. If I am not mistaken, this is not what you call a strange idea.
“If you mean it suddenly takes the function of ‘that’ in this sentence because you left out the comma, it’s just wrong.”
Could you rephrase this sentence? I wonder what you want to say.
(Thank you for your responses.)
I think there are plenty of examples in this thread showing how switching between “that” and “when” changes the meaning of the sentence. Google can probably find many more. As with most subtle usage details (in most languages), reviewing lots of examples is probably going to give you more information than trying to establish an artificial rule. Especially when native speakers are telling you that your proposed rule is incorrect.
Relative clause
Relative adverb
Restrictive relative clause
Definition of THAT
1
—used as a function word to introduce a restrictive relative clause and to serve as a substitute within that clause for the substantive modified by the clause [the house that Jack built]. [I’ll make a ghost of him that lets me — Shakespeare]
2
a : at which : in which : on which : by which : with which : to which [each year that the lectures are given]
b : according to what : to the extent of what —used after a negative [has never been here that I know of]
3
a archaic : that which
b obsolete : the person who
P.S.
‘On which’ or ‘in which’ can mean ‘when.’
I am beginning to think that ‘that’ can have the meaning of ‘where’ in a certain context does not necessarily signify that ‘that’ can be replaced by ‘when’ even in the same context.
"Buuut[sic], you seem to want to speak the way you like rather than listen to native speakers’ advice… so, go right ahead! "
You are using a very interesting language. I am more interested in writing than speaking like native speakers.
P.S.
I inserted the second-last sentence.
“I am beginning to think that ‘that’ can have the meaning of ‘where’ in a certain context does not necessarily signify that ‘that’ can be replaced by ‘when’ even in the context.”
Bingo.
Same difference. You seem very interested in writing in your own way, which is incorrect, despite several native speakers’ explanations of why it is incorrect and their alternative suggestions.
“When” can refer to time, but can also mean “in which case.”
In order to avoid ambiguity, or a completely wrong meaning, you have to notice whether the use of “when” would give your sentence a dual meaning or an incorrect meaning.
No comma and when
–restrictive relative clauses beginning with a relative adverb “when”
“It was a time when people didn’t have to lock their doors.”
“the happy days when we were together”
I feel that “when” in the first sentence cannot be replaced by “that” for some reason. I wonder whether “when” in the second sentence can be replaced by “that.”
“the happy days when we were together” <—> “the happy days that we were together”
If this is possible, you can also say that “that” can be replaced by “when” in some cases.
I think it is stronger to say “that” here. It wasn’t “when” he was at the sanatorium it was the six weeks that he was there. When for me indicates he could have been somewhere else and this was a happenstance. He spent other weeks elsewhere but these are when he was there. When opens a door into time. There is none of that in that. For the first six weeks that he was there he stayed in bed. Thank God for sanatoriums eh?
“He could remember a time when he had worked like that himself.” —Collins COBUILD ADVANCED DICTIONARY
I wonder whether the following sentences are correct.
As I do not know whether my horse is dead, I am still arguing with natitive speakers of English about how to use ‘that’ and ‘when.’ I have no intention of ‘reinventing’ the English language. I am simply interested in the usage and in improving ‘my’ English, not the English language. I welcome any comments.
I know that there are a few rules concerning how to use ‘that’ and ‘which’ as relative pronouns. I wonder whether these are applicable to the difference between ‘that’ and ‘when’ as relative adverbs.
That’ or ‘which’?
“Do you sometimes wonder whether to use that or which in a sentence? In many cases, in British English, both words are equally correct.”
Oxford Languages | The Home of Language Data
Which or That?
Which or That? | Get It Write Online
The second article suggests that you use “which” only in a non-restrictive relative clause with a preceding comma. However, in the first article, “which” is used not only in a non-restrictive clause but also in a restrictictive clause as follows:
She held out the hand which was hurt.
She held out the hand that was hurt.
I bought a new dress, which I will be wearing to Jo’s party. [non-restrictive]