I use a KVM switch (two port) with multimedia ability. One set of speakers works with both computers.
KVM switches also come in 4 port models.
Note: I'm not sure if a KVM switch will work if you only attach the speakers. I have 1 LCD monitor, one keyboard, one mouse and 1 speaker set for two computers.