I keep dedicated computers and phones for music and rarely update them. Once they are stable and if kept unconnected to the internet and other networks, refraining from updates causes no issues.
I've also gravitated towards writing my own software and using opensource alternatives rather than relying on the development efforts, as prodigal as they can be, of a lone developer. This way, I can keep the software going through the inevitable hardware upgrade.
For hardware, I no longer purchase any USB controller or audio devices that aren't "class compliant" (i.e. those that do not require drivers). This removes the dependency on driver upgrades and keeps everything running through updates.