Notes about locking in the Open-MX driver
=========================================

The endpoint has 2 main status: FREE and OK. To prevent 2 people from changing it
at the same time, it is protected by a spinlock. To reduce the time we hold the lock,
there are 2 intermediate status: INITIALIZING and CLOSING.
When an endpoint is being used, its refcount is increased (by acquire/release)
When somebody wants to close an endpoint, it sets the CLOSING status (so that
new users can't acquire the endpoint), remove it from the interface, and the
the last user will release it for real.
The spinlock is taken when opening and closing. Bottom halves acquire the endpoint
first, test the status, and release if the status is wrong.

The iface has both a kref to detect the last user and also has a number of endpoints
attached to detect when we need to force.
There's a mutex to protect this array against concurrent endpoint attach/detach.
When removing an iface (either by the user or by the netdevice notifier), the status
is set to CLOSING so that any new endpoint opener fails.
The mutex is taken only when attach/detaching endpoints, and RCU primitives are used
to protect readers concurrently accessing it.

When an iface is removed, all endpoints are scheduled for closing if necessary
(if forced) and the main iface reference is released. The last endpoint will release
the last reference and thus release the device. When this happens due to the
unregister notifier, the caller will wait for the last device reference to be
released (in rtnl_unlock()), so we can return from the detach routine earlier
as long as we guarantee that things are being closed soon.

The list of ifaces is always coherent since new ifaces are only added once
initialized, and removed in a coherent state (endpoints have been properly
detached first). It is modified using RCU primitives to protect readers from
writers modifications.
Incoming packet processing is disabled while removing an iface anyway.
So the iface may not be removed while processing an incoming packet, so
we don't need more locking and no need hold a reference on the iface either.
No need to disable bottom halves since it never scans the array of ifaces
(and the notifier callback may not be called from BH since it is interruptible).

The endpoint region list and the peer table are protected by RCU locks.
The per-endpoint region list uses call_rcu() to defer deletion of the
region after the grace period instead of having the very expensive
synchronize_rcu() in the critical path of large messages.
All other RCU users (ifaces array, per-iface endpoints array, peers table)
use synchronize_rcu() to wait for readers to exit before deleting an object,
we don't care about waiting a bit long there.
