ABSTRACT: It is often assumed the navigation is used by animals indicates landmarks to locate the goal. However, many mammals including human and insects do homing by relying on self-motion cues. This process is known as Path integration. Path integration enables desert arthropods to find back to their nest on the shortest track from any position. We consider a mathematical model of a navigation system according to experiments conducted on humans in virtual environment in which only visual cues were provided. To perform path integration, forward moving speed and the angular turning rate are implemented into a linear system of differential equations in both Cartesian and Polar coordinate systems. Because in homing by triangle completion, homing distances were biased in the experimental results by subjects, we generate some Gaussian noises by Monte Carlo simulation for our mathematical model. The main objective of this work is fitting a quantitative model to the homing trajectory endpoints of experimental data, measure the goodness of fit, and find the best values of noise parameters. Here we used an extended Nelder-Mead simplex method as an optimization method to find the optimum values of noise parameters for our mathematical model.