In the process of drilling oil and gas wells, the shrinkage and falling of wellbore walls are often caused by the expansion of mud shale water. To date, conventional additives have been unable to plug the pore throats of shale rock with nanoscale pores and thus cannot effectively solve the problem of wellbore instability encountered in shale formation drilling. In view of this situation, the idea of using nanosilica to plug the nanopore throat of mud shale is proposed to reduce its permeability and to slow water intrusion. The influences of nanosilica on the properties of brine-base drilling fluid drilling fluid are evaluated by measuring the viscosity, filtration loss and swelling of the drilling fluid. The results of laboratory experiments show that the improvement in drilling fluid properties can only be achieved on the basis of salt resistance; that is, a salt-resistant soil slurry should be used. A concentration of 1–5% nanosilica can improve the viscosity of the drilling fluid by increasing the internal friction between particles. However, nanosilica materials are sensitive to salt concentration. Nanosilica particles can be deposited on the surface of a filter cake to block the pores of the filter paper, and the filtration loss reduction rate can reach 40.2%. Blocking the pores of the clay plays a role in preventing the clay from absorbing water and expanding. The optimal addition amount of silica is 3%, and its salt resistance can reach 16%. Considering the experimental results of filtration loss, swelling amount and cost performance, 3%NP + 4%NaCl + SWM-B is selected as the optimal formula. The results of this study can be applied to effectively improve the phenomenon of wellbore instability during drilling in shale formations, and it has important application value.