Abstract This paper investigates the influence of the concentration and particle size of h-BN nanoparticles in a nanofluid on the surface integrity of 304 austenitic stainless steel during turning, focusing on the cutting force, friction coefficient, cutting temperature, surface roughness, surface residual stress, work hardening capacity, and 3D surface topography. The results show that, compared to dry cutting, the addition of 3 wt.% h-BN nanofluid can reduce the friction coefficient on the rake face by 38.9%, lower the cutting temperature by 43.5%, decrease the surface roughness by 53.8%, decrease the surface residual stress by 61.6%, and reduce the work hardening degree by 27.5%. Two-dimensional profiles and the 3D surface topography display a more balanced peak–valley distribution. Furthermore, by studying the effect of different h-BN particle sizes in nanofluids on the surface integrity of the machined workpiece, it was found that nanoscale particles have a greater tendency to penetrate the tool–chip interface than submicron particles. Moreover, the h-BN particles in the nanofluid play a “rolling effect” and “microsphere” effect, and the sesame oil will also form a lubricating oil film in the knife-chip contact area, thereby reducing the friction coefficient, reducing the cutting force, and improving the machining surface quality. Keywords: nanofluid; minimum-quantity lubrication; hexagonal boron nitride; surface integrity