Thanks for raising this @mzagmajster - this is a very long running feature request:
Summary idea thread: Implement roles (in addition to access levels) for users
GitHub issues relating to this topic:
2016: Finer control over a role's access to Mautic items · Issue #3097 · mautic/mautic · GitHub
2018: Grouping Users by role, so they are able to share content · Issue #6162 · mautic/mautic · GitHub
2018: Add User Groups for more granular Permission Controls... · Issue #6642 · mautic/mautic · GitHub
I think this would also really compliment the work we have going on in the resource management initiative (currently stalled waiting on developer) to organise resources by project - the two would indeed be very powerful features for the marketer in isolation but together (eg assigning a project to a department and having them inherit all the stuff within the project) would be super awesome for marketers I think.
So yes, I think this is a feature that would be great to investigate more and thank you for raising it! 
.
We would need to be mindful as you have mentioned already about how it interacts with the Contact Owner - maybe that would be something separate to the departments having access. Access doesn’t always mean ownership, but ownership would necessitate access.
We would also need to think it through carefully as regards the permissions side of things or I can see it becoming ridiculously confusing and complex!
Side note:
It might also be relevant to raise this suggestion which could allow us to deploy something like this under a beta testing flag, so folks could choose to turn it on for a couple of minor releases before it became available by default. It’s something I proposed some time ago but we haven’t got around to doing yet. With the new plugins / composer architecture I think it is probably easier to do this nowadays.