Recently hyper-graph learning gains increasing attention in medical imaging area since the hyper-graph, a generalization of a graph, opts to characterize the complex subject-wise relationship behind multi-modal neuroimaging data. However, current hyper-graph methods are re-strained with two major limitations: (1) The data representation encoded in the hyper-graph is learned only from the observed imaging features for each modality separately. Therefore, the learned subject-wise relation-ships are neither consistent across modalities nor fully consensus with the clinical labels or clinical scores. (2) The learning procedure of data representation is completely independent to the subsequent classification step. Since the data representation optimized in the feature domain is not exactly aligned with the clinical labels, such independent step-by-step workflow might result in sub-optimal classification. To overcome these limitations, we propose a novel dynamic hyper-graph inference frame-work. Working in a semi-supervised manner, it iteratively estimates and adjusts the subject-wise relationship from multi-modal neuroimaging data until the learned data representation (encoded in the hyper-graph) achieves the largest consensus with the observed clinical labels and scores. It is worth noting that our inference framework is also flexible to integrate classification (identifying individuals with neuro-disease) and regression (predicting the clinical scores). We have demonstrated that the performance of our proposed dynamic hyper-graph inference framework renders more accurate diagnosis result in identifying MCI (Mild Cognition Impairment) subjects and the fine-grained recognition of different progression stage of MCI compared to conventional counterpart methods.