The probability density approach based on the response-excitation theory is developed for stochastic simulations of non-Markovian systems. This approach provides the complete probabilistic configuration of the solution that enables a comprehensive study of stochastic systems. By using functional integral methods we determine a computable evolution equation for the joint response-excitation probability density function (REPDF) of stochastic dynamical systems and stochastic partial differential equations driven by colored noise. We establish its connection to the classical response approach and its agreement to the Dostupov-Pugachev equations (Dostupov, 1957) and the Malakhov-Saichev equations (Gurbatov et al, 1991). An efficient algorithm has been proposed by using adaptive discontinuous Galerkin method and probabilistic collocation method combined with sparse grid. For high-dimensional REPDF systems, we develop the algorithms concerning high-dimensional numerical approximations, namely, separated series expansion and the ANOVA approximation. These methods reduce the computational cost in high-dimensions to several low-dimensional operations. Alternatively, reduced order PDF equations are obtained by using the Mori-Zwanzig framework and conditional moment closures, which establish a preliminary work of goal-oriented PDF equations. Finally, we demonstrate the effectiveness of the proposed numerical methods to various stochastic systems including the tumor cell growth model, chaotic nonlinear oscillators, advection reaction equation, and Burgers equation. The second part of the thesis focuses on simulations of multi-scale stochastic systems. The Karhunen-Loeve expansion is extended to characterize multiple correlated random processes and local decomposed random fields. We then propose interface conditions based on conditional moments and PDE-constrained optimization that preserve the global statistics while propagating uncertainty. Finally, the decomposition algorithm is recast to couple distinct PDF models including the REPDF system. Advisors/Committee Members: Karniadakis, George (Director), Rozovsky, Boris (Reader), Venturi, Daniele (Reader), Sapsis, Themistoklis (Reader).