What you hear is, the room that has been energised by the air movements of the speakers.
So your perception is correct! Always start with the speakers, then the amplifier as it doesn't just drive the speakers, it is more of a coupling. The concept of get the best source you can afford, came about in the analogue days of TT's and Tape Decks. Where the more you spent the better quality you seemed to get, so that legend remains in the audiophile psyche. Although there always was that "one piece" of equipment that broke that rule. With digital sources, the money spend is a lot less critical, as it has really democratised music reproduction. Any money you spend on improving the room's acoustics, will always exceed any gains you wiil get from other expensive tweaks.