If a new environment favored lower (or higher) aggressiveness in males , a Y-chromosome that induced lower (or higher) aggressiveness would take off. And since different Y chromosomes do indeed affect the level of aggressiveness in mice [which I just found out], possibly by affecting testosterone production – this mechanism is plausible. [. . .]A 2009 study of 156 Pakistanis found an association opposite what one might expect, but I doubt it would replicate in a larger sample:
Fortunately for all concerned, the selective value of aggressiveness, etc. has been the same for all human populations forever and ever, before and after the development of agriculture. Otherwise you might see weirdly rapid expansions of particular Y-chromosome haplogroups – common, yet only a few thousand years old.
Five Y haplogroups that are commonly found in Eurasia and Pakistan comprised 87% (n=136) of the population sample, with one haplogroup, R1a1, constituting 55% of the sampled population. A comparison of the total and four subscale mean scores across the five common Y haplogroups that were present at a frequency > or =3% in this ethnic group revealed no overall significant differences. However, effect-size comparisons allowed us to detect an association of the haplogroups R2 (Cohen's d statistic=.448-.732) and R1a1 (d=.107-.448) with lower self-reported aggression mean scores in this population.A PhD thesis published this year ("The Y chromosome in cardiovascular disease") looks at reasonably large samples of Polish men and evidently finds no associations between Y haplogroup and sex hormones or aggression:
Though no analysis of the human Y chromosome has ever been completed in the context of these \male-relateda or \sex-specifica phenotypes, evidence from animal models supports a hypothetical role for the Y chromosome in regulation of both aggression and sex hormones. Firstly, the Y chromosome has been repeatedly implicated in aggression in murine models; Gatewood et al. found that female mice carrying an SRY-deleted Y chromosome had significantly higher aggression levels than wild-type female mice, similar to those found in males (Gatewood et al. 2006). Similarly, the Y chromosome was associated with sex steroids through its function in sex determination (Wilhelm et al. 2007).(More interestingly, while I haven't looked at the thesis closely it does apparently confirm in Polish men an association between Y haplogroup I and cardiovascular disease, which I hadn't taken too seriously when it was previously reported in British men.)
To identify the mechanism of association between haplogroup I of the Y chromosome and CAD, the effect of this haplogroup on \sex-associateda and \sex-specifica phenotypes was investigated in a series of experiments highlighted in this chapter. No relationship between the Y chromosome and any facet of aggression or sex steroid was identified. These data indicate that these traits are unlikely to drive the association between haplogroup I and increased predisposition to CAD.
Although a lack of convincing evidence that major extant European Y haplogroups are differentially associated with aggression doesn't rule out the possibility that Y chromosomes associated with, e.g., lower aggressiveness constituted a larger share of European male lineages in the past, I doubt this possibility as an explanation for the recent expansion of haplogroups like R1b and R1a.