Fetch all users and Groups from a site collection using c# client Object model

I want to fetch all groups and users (outside of the the sharepoint group) from a Site collection using C# client object model. Currently i am bale to fetch all the groups and users inside those groups. But if i open Site permissions, and if any users are given permission directly (without including in any groups), how do i retrieve these user's? Please suggest.