Aphasia is a common type of acquired language impairment resulting from dysfunction in specific brain regions. Analysis of narrative spontaneous speech, e.g., story-telling, is an essential component of standardized clinical assessment on people with aphasia (PWA). Subjective assessment by trained speech-language pathologists (SLP) have many limitations in efficiency, effectiveness and practicality. This paper describes a fully automated system for speech assessment of Cantonese-speaking PWA. A deep neural network (DNN) based automatic speech recognition (ASR) system is developed for aphasic speech by multi-task training with both in-domain and out-of-domain speech data. Story-level embedding and Siamese network are applied to derive robust text features, which can be used to quantify the difference between aphasic speech and unimpaired one. The proposed text features are combined with conventional acoustic features to cover different aspects of speech and language impairment in PWA. Experimental results show a high correlation between predicted scores and subject assessment scores. The best correlation value achieved with ASR-generated transcription is .827, as compared with .844 achieved with manual transcription. The Siamese network significantly outperforms story-level embedding in generating text features for automatic assessment.