The root cause was configured auto-snapshots of the filesystem for backups took to much disk space (but not all though), which for some reason caused ZFS to put parts of the server into read only mode. We disabled the backups for now and restarted the server.
The final solution will be to free more disk space.