Today short before ending business hours I was noticed that there is a problem with a server system (domU). Login with unprivileged user was possible but using "su" didn't worked, also login in as root via privkey failed. Fortunately I was able to connect via xen console and login via tty. Looking into the bash history the reason revealed quickly:
4979 2013-02-01 15:03:39 cd /var/www/test.org/data/galleries/
4980 2013-02-01 15:03:43 chown www-data:www-data -R /var/www/test.org/data/galleries/
4981 2013-02-01 15:04:36 ls -la
4982 2013-02-01 15:04:46 ls -la
4983 2013-02-01 15:04:54 chown www-data:www-data -R /*
4984 2013-02-01 15:07:42 chown www-data:www-data -R /var/www/test.org/data/galleries/
4985 2013-02-01 15:36:55 chown www-data:www-data -R /var/www/test.org/data/galleries/
This made my day (and maybe parts of the rest of the weekend).
For recovery our 1st Level mounted the domU-fs on the dom0 to '/tmp/recover' and did:
2131 2013-02-01 21:29:28 cd /tmp/recover
[...]
2142 2013-02-01 21:31:17 rm -r lib64/
The experienced reader may see the problem:
# ls -lad lib64
lrwxrwxrwx 1 root root 4 Jun 28 2011 /lib64 -> /lib
So also the dom0 was knocked out ... what a funny evening (and maybe night). Maybe our staff looked similar like here.