Free space optical communication is prominent technique and premier alternative to RF
communication. Nowadays, Researchers are giving attenuation to the hybrid radio over free space optical
communication because of long reach and MDM also getting attention due to high capacity catering
potential. In this research work, 2 x 40 Gbps-40 GHz Ro FSO system is proposed using different
modulation formats such as modified duo-binary return to zero (MDRZ), non return-to-zero (NRZ),
return-to-zero differential phase shift keying (RZ-DPSK), alternate mark inversion (AMI) and non return to-zero differential phase shift keying (NRZ-DPSK). For the MDM, two Hermite Gaussian (HG) modes
are used under the effects of different weather instabilities such as light, moderate and heavy fog. Input
parameter such receiver antenna aperture diameter is varied and it is perceived that the increase in
aperture diameter of receiver, Q factor and BER also improve. Results revealed that MDRZ performs best
followed by NRZ-DPSK, RZ-DPSK, AMI and NRZ schemes.