Safety and reliability are the major challenges to face for the development and acceptance of hydrogen technology. It is therefore crucial to deeply study material compatibility, in particular for tribological components that are directly in contact with hydrogen. Some of the most critical parts are sealing materials that need increased safety requirements. In this study, the fretting behavior of several elastomer materials were evaluated against 316L stainless steel in an air and hydrogen environment up to 10 MPa. Several grades of cross-linked hydrogenated acrylonitrile butadiene (HNBR), acrylonitrile butadiene (NBR) and ethylene propylene diene monomer rubbers (EPDM) were investigated. Furthermore, aging experiments were conducted for 7 days under static conditions in 100 MPa of hydrogen followed by rapid gas decompression. Fretting tests revealed that the wear of these compounds is significantly affected by the hydrogen environment compared to air, especially with NBR grades. After the aging experiment, the friction response of the HNBR grades is characterized by increased adhesion due to elastic deformation, leading to partial slip.