Callithrix Erxleben, 1777 has six species, all endemic to Brazil. The distribution of Callithrix species is closely associated with the Atlantic Forest. Callithrix jacchus (Linnaeus, 1758) and Callithrix penicillata (É. Geoffroy, 1812) have the largest natural geographical distribution within the genus. They are found in the Atlantic Forest and Caatinga of northeastern Brazil and in the Cerrado of central and northeastern Brazil (De Vivo 1991, Rylands et al. 1996). They are phylogenetically very close, and it has been hypothesized that their most recent common ancestor lived about 700 thousand years ago, in the Atlantic Forest, Cerrado, and Caatinga. A subsequent vicariant speciation event isolated the ancestor of C. penicillata in the Cerrado or Caatinga (Buckner et al, 2014Malukiewicz et al. 2014.

Species of Callithrix are commonly called marmosets. The range of Callithrix species is allopatric, with some species contacting at the limits of their distribution. However, the ranges of natural species are being altered due to habitat destruction and to anthropogenic introduction of marmoset species outside their natural geographical bounds. As a result of such anthropogenic alterations, C. jacchus and C. penicillata are often found in sympatry with several other Callithrix species and in the natural ranges of other primates (Rylands et al. 1993, 2009, Ruiz-Miranda et al. 2000).

Callithrix jacchus and C. penicillata are found in the state of Rio de Janeiro, Brazil, both in forested and disturbed areas. Their introduction and settlement in the state are the result of illicit domestic and international trafficking of primates. Although the history of their introduction into the coastal lowlands is uncertain, the distribution of marmosets is increasing towards the north of the state of Rio de Janeiro in the lowlands at an estimated rate of 1.2 km per year (Ruiz-Miranda et al. 2000). There, they are found in forest fragments of costal lowlands where the golden lion tamarin, Leontopithecus rosalia (Linnaeus, 1766), is naturally distributed. The interaction between C. jacchus and C. penicillata and the native populations of golden lion tamarins is problematic because the ecology of these primates is similar, which may lead to competition for food and territory and disease transmission (Ruiz-Miranda et al. 2000.

Callithrix jacchus and C. penicillata are differentiated by the colors of the body pelage and of the auricular tufts, and by the insertion of tufts in the ear. C. penicillata has black auricular tufts arranged in front of the ear (pre-auricular) whereas C. jacchus has white auricular tufts arranged around the ear (circum-auricular) (Hershkovitz 1977). Most studies on the identification of hybrids consider only the pelage and mitochondrial DNA, excluding cranial morphological characters (e.g. Alonso et al. 1987, Fuzessy et al. 2014).

This study compares genetic hybrids from the coastal lowland with five species of the genus Callithrix and other hybrids from Rio de Janeiro’s metropolitan region, in order to evaluate morphological and genetic diversity, identify the species involved in hybridization, and verify the geographic structure. To this end, we performed cranial and pelage morphological analyses and molecular phylogenetic estimation using the cytochrome b mitochondrial gene (MT-CYB).

The sample studied herein comprised 39 individuals: ten Callithrix hybrids from two municipalities in the coastal lowlands of the state of Rio de Janeiro, Silva Jardim (N = 9) and Rio das Ostras (N = 1); and C. penicillata (N = 14) and C. jacchus (N = 15) from localities near the type localities. This sampling strategy ensured that we handled samples of each species separately, thereby avoiding the presence of hybrids. This strategy also limited the sample size. The pure specimens analyzed were identified by the pelage description provided by Hershkovitz (1977). Hybrids were identified by the presence of intermediate characters. Voucher numbers for the specimens analyzed are available in the Appendix 1.

Auricular tuft color and disposition, and general pelage color of all individuals were analyzed to estimate the variation in pelage color and specific patterns for each species and for the hybrids.

A stereoscopic microscope was used to analyze the qualitative characters of C. jacchus and C. penicillata skulls. Differences between species and between each species and the hybrids were identified.

Ten linear cranial measurements ’were taken from the marmosets, with a digital caliper (mm). The first six were defined by Natori (1994) and De Vivo (1991) and the last four in this study: (PL) prosthion to lambda, (EE) euryon to euryon, (iFO) inside frontomalare orbitale to frontomalare orbitale, (BB) bicondylar breadth, (CM) mesial surface of the left upper canine to distal surface of the left second upper molar, (ZA) zygomatic arch breadth, (oFO) outside frontomalare orbitale to frontomalare orbitale, (MS) mandibular symphysis height, (lFM) foramen magnum length, and (bFM) foramen magnum breadth.

To analyze the morphological differences between the studied Callithrix species and the hybrids, we calculated the mean, standard deviation, maximum and minimum values of morphological measurements described above.

Student’s t-test and one-way ANOVA were used to identify differences between species and between each species and the hybrids. Principal component analysis (PCA) was performed to reveal patterns of variation between species and hybrids, and to visualize differences among them. Discriminant function analysis (DFA) was used to verify if the a priori classification of each individual as C. jacchus, C. penicillata, or hybrid using qualitative characters was correct. Analyses were performed in Statistica 8 (Statistica Software Inc.) and R 3.2.4.

DNA samples were obtained from tissue samples extracted from hybrid specimens and from the species Callithrix penicillata, C. aurita, C. kuhlii, Mico rondoni, and Saguinus mystax (Table 1). The latter two were used as outgroups. The DNA was extracted following a phenol-chloroform protocol (Sambrook and Russell 2001). Primers for L14724 (Irwin et al. 1991) and Cytb rev (Casado et al. 2010) were used to amplify MT-CYB. The PCR product of the MT-CYB gene was purified and sequenced using the same PCR primers and the internal primers Sot in1 and Sot in2 (Cassens et al. 2000); Alo aot F and Alo aot R (Menezes et al. 2010); and Citb alo (Bonvicino et al. 2001). The product was labeled with XL and BigDye Terminator v3.1 Cycle Sequencing Kit (Applied Biosystems). Sequencing was carried out in an ABI 3130 xl platform. To improve our data, sequences of C. jacchus (accession numbers: AF295586 and AY434079), C. penicillata (accession number: KR817256.1) and C. geoffroyi (accession number: HM368005) were obtained from GenBank online database (www.ncbi.nlm.nih.gov/genbank).

The sequences were analyzed and edited in the software ChromasPro (Mccarthy 1998) and manually aligned in MEGA 5.0 (Tamura et al. 2011).

Genetic distances were estimated with complete deletion using the Kimura 2-parameter model. The MEGA 5.0. Model Generator 0.85 (Keane et al. 2006) identified the best-fitting model for nucleotide substitution using second-order Akaike Information Criteria (AIC) (Akaike 1973).

DNAsp 5 (Librado and Rozas 2009) was used to estimate haplotype and nucleotide diversity. NETWORK was used to reconstruct a median-joining (MJ) network (Bandelt et al. 1999).

Maximum likelihood (ML) and Bayesian inference (BI) phylogenetic trees were built. The ML analysis was inferred using a TN93 + I nucleotide substitution model (Tamura and Nei 1993) and the bootstrap analysis was based on 1000 replicates with PhyML (Guindon et al. 2010). MRBAYES 3.2 (Huelsenbeck and Ronquist 2001) was used to build the Bayesian tree using a TN93 + I model.

Hybridization has been consistently documented in primates. However, the effects of natural and anthropogenic hybridization on biodiversity remain unclear. Differentiating between these two types of hybridization is a challenge in evolution and conservation studies (Malukiewicz et al. 2014). Previous reports showed that hybrids have intermediate characters between the parental species (Hershkovitz 1977, Alonso et al. 1987, Fuzessy et al. 2014, a finding that is partially supported by results of the current study. One possible explanation for such finding is the fact that the parental species are phylogenetically very close. (Tagliaro et al. 1997, Buckner et al. 2014. Mallet (2005) argued that hybridization occurs in approximately 10 percent of the mammalian species, usually between groups that diverged more recently (1 to 2 million years). C. jacchus and C. penicillata are species that diverge very recently (Malukiewicz et al. 2015).

We observed that the auricular tufts (one of the main characters used to identify marmosets) of the hybrids do not match the description of any species of the genus. In the hybrids the tufts were white or gray, arranged anterior and laterally to the ear and were broken in some portions, a mosaic that may result from several generations of hybridization. In the study of Alonso et al. (1987), involving hybrids between C. jacchus and C. penicillata in natural hybrid zones, the recorded color pattern and shape of the auricular tufts suggest that there is a reproductive isolation mechanism between these species, since there was little penetration of C. jacchus characters in the C. penicillata population and vice-versa. Individuals with pure parental phenotypes were absent in the hybrid groups of that study, as was observed in the present one.

Hybrids of mixed ancestry between two marmosets species (C. penicillata and C. geoffroyi) had greater morphological variation compared with individuals of pure ancestry (Fuzessy et al. 2014), which could possibly explain the results found for the hybrid individuals in our study. Malukiewicz et al. (2014) agreed that the current situation of coastal lowland marmosets is the result of multiple introductions and that there some new genetic variations caused by new introductions. These multiple introductions may not only be from pure parental individuals but also from hybrid individuals, resulting in crosses between pure and mixed ancestries that give rise to highly variable phenotypes. The hybrids analyzed herein showed a large variation in body color, with a predominance of light brown regions throughout the body (detected even in the tail). This has not been previously observed in either of the two studied species and may be related a transgressive segregation. Studies on hybrid populations have occasionally reported the presence of phenotypes that are extreme relative to those of either parental line (Rieseberg and Ellstrand 1993, Cosse et al. 1995). The generation of these extreme phenotypes is referred to as transgressive segregation, a phenomenon specific to segregating hybrid generations and refers to the fraction of individuals that exceed parental phenotypic values in either a negative or positive direction (Rieseberg et al. 1999).

Only one qualitative character differed between C. jacchus and C. penicillata: the presence/absence of a prolongation of the maxilla, after the last molar. Hybrids had the space, being therefore similar to C. jacchus. This is an unprecedented observation in the taxonomic literature, since authors report the skulls of C. jacchus and C. penicillata as being very similar (Garbino 2015, Natori 1994).

Three measurements explained most of the variation observed in the PCA. When specimens are plotted along PC1 and PC2, the distribution is somewhat scattered. However, there are some trends according to the species and hybrids. An overlap exists between C. penicillata and C. jacchus, which may be explained by the great similarity between their skulls. Similarly, there is an overlap between hybrids and C. jacchus, indicating that hybrids are more related with C. jacchus than with C. penicillata. This result differs from that obtained by the univariate analysis.

The results of the DFA show that Wilk’s lambda was relatively small (0.065998) and the approximate F value was high (6.0744, p < 0.0001), corroborating the a priori classification.

Results of the phylogenetic analysis supported the monophyly of Callithrix, as expected, based on previous phylogenies (Tagliaro et al. 1997, Perelman et al. 2011, Garbino 2015). In the current study, C. jacchus is the sister group of C. penicillata, and these two are the most recent split in the genus. Hybrids were grouped in a clade with C. penicillata, suggesting that this species is the one involved in the maternal lineage of the individuals, being a direct or indirect parental species. Where C. penicillata was not a direct parent, crosses between lineages of hybrids with C. penicillata maternal origin may have occurred. Both cases suggest that males preferentially mate with females from the C. penicillata lineage.

Mate choice by males is primarily associated with mate availability and with variation in female quality. If males engage in paternal care, as is the case in Callithrix, the average number of available females is likely to be high compared to the capacity of males to mate with them. In this situation, males are less likely to be able to mate with all available females, rejecting some of them. If the benefit of mating with specific females exceeds the cost of assessing them, mate choice can evolve (Edward and Chapman 2011). Although fertile hybrids will be generated between the two species, some matings may result in less viable hybrids than others, influencing the process of mate choice (Coimbra-Filho et al. 1993). The relationship found between the hybrids and C. penicillata in this study suggests that hybrid males prefer C. penicillata females. This may be because hybrids that result from mating with C. penicillata females are more viable than those that result from mating with C. jacchus females.

Primates recognize potential mates (members of the same species or not) based on visual (mainly on the face), acoustic, olfactory, and other sensorial cues. Different patterns of facial color and auricular hairs have diagnostic value for each species in the taxonomy of Callithrix. One conspicuous morphologic character that distinguishes these four species is the coloration of the auricular tufts (Cavalcanti and Langguth 2008).

Cavalcanti and Langguth (2008) suggested that there is an isolation mechanism based on head color, since when two different species are together, one responds to the other with significantly less frequency than to its own species. However, this is not always the case, suggesting that, in spite of the conspicuous differences in facial coloration patterns, the evaluator species recognizes individuals of the other, “cue-bearing” species as potential sexual competitors. During speciation events in Callithrix, reproductive isolation mechanisms did not necessarily appear simultaneously in all the presently recognized species. Populations of each species have a different history. C. jacchus and C. penicillata are very closely-related and have split recently from a common ancestor. Thus, it is possible that reproductive isolation is not complete in this case. Although the primate literature is relatively rich in studies of sexual selection and mate preference in hybrid zones (Shurtliff 2013), little attention has been given to these topics in recent decades. A study on howler monkeys (Alouatta palliata and A. pigra) showed that hybridization and subsequent backcrossing are directionally biased, probably producing only fertile hybrid females and inviable or infertile males. This suggests that a process of mate choice may occur for Neotropical primates in hybrid zones (Cortés-Ortiz et al. 2007).

The low genetic distance between hybrids revealed that they are genetically very close, corroborating the hypothesis of multiple introductions suggested by the morphological results and the short passage of new genetic variations.

Haplotype H1 was found in most of the samples, containing both CL and MR individuals and precluding the existence of a geographical structure. A possible explanation is that introductions of marmosets into the coastal lowlands occurred recently with animals from the metropolitan region. De Morais et al. (2008) argued that the history of the introduction of marmosets in the coastal lowlands of Rio de Janeiro is uncertain and emphasized the occurrence of two major releases of marmosets, seized by regulatory agencies (> 60 animals), between 1983-1987, in regions close to the study area. Each of the haplotypes, H2, H3, and H4 was found in only one CL hybrid, and haplotype H5 was found in two CL individuals. This diversity in a single region may be the result of multiple introductions, with the arrival of new individuals from different localities. Haplotype H6 included one C. penicillata and one MR individual, but the fact that haplotype H1 also included an MR individual indicates that H6 is not characteristic only of the MR.

The network analysis showed close a relationship between the hybrids and C. penicillata, confirming the result from the ML and BI analyses. At least one variable site separates each haplotype, which may reflect differences due to multiple introductions or a polymorphism in the population. Twenty-one variable sites were observed, with eighteen transitions and three transversions. Haplotype diversity among hybrids and C. penicillata was high (0.7582), whereas nucleotide diversity was low (0.00537). Low nucleotide diversity may be explained by the founder effect, whereby an introduction is followed by population growth. The several haplotypes also had low diversity, possibly due to multiple anthropogenic introductions that continue to occur. The low level of genetic diversity both in the CL and between the CL and the MR suggests a recent history of population expansion, most likely associated with introductions.

These issues reflect the need to perform more detailed studies concerning hybridization and the development of morphological characters in order to obtain a better understanding of the evolution of Callithrix.