Artificial intelligence (AI) technology has recently been introduced to dentistry. AI-assisted cephalometric analysis is one of its applications, and several commercial AI services have already been launched. However, the performance of these commercial services is still unclear. This study aimed to determine whether commercially available AI cephalometric analysis can replace manual analysis by human examiners. Eighty-four pretreatment lateral cephalograms were traced and examined by two orthodontists and four commercial AIs, and 13 commonly used cephalometric variables were calculated. Then, the Bland-Altman analysis was conducted to evaluate systematic and random errors between examiners. The interchangeability of an AI was determined if the random errors of the AI were smaller than the clinically acceptable limits derived from the random errors between human examiners. Finally, the inter-examiner reliability index was calculated, and Cohen's kappa was determined to assess the actual classification reliability of each examiner. The systematic errors of the AIs were clinically insignificant in general. However, the random errors of the AIs were approximately twice those of human examiners, which did not satisfy the interchangeability condition. Furthermore, even though the reliability indices of the AIs were in the good-to-excellent range, their classification reliability was unacceptable. Commercial AI is still at a level that makes it challenging to replace manual landmarking by human experts. Thus, a human examiner's landmark position review is mandatory when using commercial AIs.