In this paper, we propose to parallelize a Hybrid Genetic Algorithm (HGA) on Graphics Processing Units (GPUs) which are available and installed on ubiquitous personal computers. HGA extends the classical genetic algorithm by incorporating the Cauchy mutation operator from evolutionary programming. In our parallel HGA, all steps except the random number generation procedure are performed in GPU and thus our parallel HGA can be executed effectively and efficiently. We suggest and develop the novel pseudo-deterministic selection method which is comparable to the traditional global selection approach with significant execution time performance advantages.We perform experiments to compare our parallel HGA with our previous parallel FEP (Fast Evolutionary programming) and demonstrate that the former is much more effective and efficient than the latter. The parallel and sequential implementations of HGA are compared in a number of experiments, it is observed that the former outperforms the latter significantly. The effectiveness and efficiency of the pseudo-deterministic selection method is also studied.