This study was carried out with an objective to develop a 3D simulation methodology for rotary engine combustion study and to investigate the effect of recess shapes on flame travel within the rotating combustion chamber and its effects on engine performance. The relative location of spark plugs with respect to the combustion chamber has significant effect on flame travel, affecting the overall engine performance. The computations were carried out with three different recess shapes using iso-octane (C8H18) fuel, and flame front propagation was studied at different widths from spark location. Initially, a detailed leakage study was carried out and the flow fields were compared with available experimental results. The results for first recess with compression ratio 9.1 showed that the flow and vortex formations were similar to that of actual model. The capability of the 3D model to predict the combustion reaction rate precisely as that of practical engine is presented with comparison to experimental results. This study showed that the flame propagation is dominant toward the leading apex of the rotor chamber, and the air/fuel mixture region in the engine midplane, between the two spark plugs, has very low flame propagation compared to the region in the vicinity of spark. The air/fuel mixture in midplane toward the leading apex burns partially and most of the mixture toward the trailing apex is left unburnt. Recommendations have been made for optimal positioning of the spark plugs along the lateral axis of the engine. In the comparison study with different recess shapes, lesser cavity length corresponding to a higher compression ratio (CR) of 9.6 showed faster flame propagation toward leading side. Also, mass trapped in working chamber reduced and developed higher burn rate and peak pressure resulting in better fuel conversion efficiency. Third recess with lesser CR showed reduced burn rates and lower peak pressure.