As transition into an era of data generation and collection, empirical summaries in the classical constitutive model of granular material are less able to take full advantage of the increasingly larger data sets. In this work, the deep learning (DL) method enabled mappings between high dimensional stress, strain spaces, and interaction structures among particles. Considering on current difficulties of predicting the evolution of macroscopic mechanical response, a DL based constitutive model of granular material composed of modified LSTM cell was proposed to link the macro-mechanism to microstructure properties and give consideration to the history-dependency. The hidden state of the modified LSTM unit can be initialized in line with the current state of the granular material, therefore the history-dependency of the granular material can be better combined with the memory property of the LSTM unit. Results collected from DEM simulations on samples with different particle size distributions (PSD), initial void ratio, and initial confining pressure under various loading paths are fed into the model. This model exhibits good generalization ability and high prediction accuracy in various different situations.