We present a symmetric physical layer based secret key generation scheme for Point-to-Point Optical Link (PPOL) communication by exploiting Polarization Mode Dispersion (PMD) as a random and inimitable channel characteristic. The randomness and security strength of generated cryptographic keys based on PMD is significantly high. In this paper, we present that random modulation of a probe signal caused by PMD in a high-speed data communication network (40Gb/s and 60Gb/s) is reciprocal with average Pearson correlation coefficient of 0.862, despite the presence of optical nonlinearities, dispersion, and noise in the system. 128-bit symmetric cryptographic key has been successfully generated using the proposed scheme. Moreover, PMD based encryption keys passed the National Institute of Standards and Technology (NIST) tests. We have shown through simulations with a 50km link that, with optimal key generation settings, symmetric keys can be generated with high randomness (high P-values for NIST randomness tests) and with sufficient generation rates (>50%). Furthermore, we considered an attack model of a non-invasive adversary intercepting at 10km into the link and found that the generated keys have high average key bit mismatch rates (>40%).