Support matrix machine (SMM) is a type of classification model that directly processes matrix-type data and serves as a generalization of support vector machine. Sparse support matrix machine achieves feature selection by using the L1 norm of the matrix instead of the F norm. However, studies have shown that nonconvex regularization terms generally yield more accurate approximations, thereby enhancing the model’s feature selection capability. To this end, this paper proposes a novel low-rank and sparse support matrix machine model incorporating a nonconvex regularization term. Built upon SMM, this approach replaces the nuclear norm of the matrix with a row-wise group sparsity regularizer induced by a nonconvex Geman function. This achieves sparsity while simultaneously fostering low-rank properties in the matrix, leading to improved classification performance. An improved optimization algorithm, built upon the Majorization-Minimization method, is proposed to solve this nonconvex optimization problem. Numerical experiments demonstrate that the proposed algorithm is more efficient than benchmark algorithms, and the introduction of a nonconvex regularizer can achieve better classification performance.