MOdified Gravity (MOG) is a covariant modification of Einstein's general relativity. This theory is one of the current alternatives to dark matter models. We describe dynamics of collisionless self-gravitating systems in the context of MOG. By studying the weak field approximation of this theory, we derive the equations governing the dynamics of the self-gravitating systems. More specifically, we consider the Jeans instability for self-gravitating fluid and stellar systems, and derive new Jeans mass limit $\tilde{M}_J$ and wave-number $\tilde{k}_J$. Furthermore, considering the gravitational instability in star forming regions, we show that MOG has not a significant difference with general relativity on this astrophysical scale. However, at larger scales such as intergalactic space MOG may lead to different galaxy and structure formation processes.

Keywords