Nowadays GPUs are vastly used in both desktop and HPC systems for other purposes than graphics functions because of their high capabilities in running massively parallel programs called kernels. Kernels may require different types of resources, and may simply be classified into either memory-bound or compute-bound category. Concurrently running of kernels from different classes may help achieving better performance and also resource utilization. We propose a runtime framework above CUDA layer to run independent kernels concurrently by splitting CTAs into small ones such that most of the GPU resources are being fully utilized and the makespangets reduced. The framework is also responsible for online classifying an input kernel into one of the previously learnt classes to know about its resource requirements. It then chooses which two or more kernels and how many CTAs from each to run concurrently

Keywords