A water-lubricated rubber stern bearing causes frictional vibration and noise under extreme conditions, severely threatening the survivability and concealment performance of underwater vehicles. To investigate the mechanism of friction-induced vibration and noise, a four-degree-of-freedom nonlinear dynamic model that included bearing support vibration, frictional bearing vibration and torsional shaft vibration was proposed. The velocity-dependent Stribeck friction model described the dynamic friction characteristics between the rubber bearing and stern shaft. Nonlinear system stability was examined using the Lyapunov indirect method and the transient dynamic responses determined by the Runge-Kutta method. Meanwhile, a high-speed in situ measurement experimental study was adopted to explore the mechanism of frictional and torsional vibrations in a water-lubricated bearing-shaft system. Results showed that nonlinear friction excitation was the fatal cause of friction-induced vibration in bearing-shaft system. Under different friction excitations, the coupling vibrations formed between the bearing and bearing support were different, resulting in two main modes of friction-induced vibration: “Chatter” and “Squeal.” The frictional vibration of the bearing and torsional vibration of the stern shaft were tightly coupled. These trends and results might provide a theoretical basis for further research on frictional vibration and wear control in water-lubricated rubber bearing-shaft systems.