Neural networks (NNs) normally do not allow any insight into the reasoning behind their predictions. We demonstrate how influence functions can unravel the black box of NN when trainedto predict the phases of the one-dimensional extended spinless Fermi-Hubbard model at half-filling.Results are the first indication that the NN correctly learns an order parameter describing thetransition. Moreover, we demonstrate that influence functions not only allow to check that the network trained to recognize known quantum phases can predict new unknown ones but even disclosesinformation about the type of phase transition.