We conducted several pilot studies to assess the psychometric prop- erties and validity of the test. The first pilot involved 418 senior under- graduate students with experience operating virtually in a team. This pilot was used to assess the psychometric properties of the test (reliability, ac- ceptable levels of item difficulty, and acceptable item-total correlations). As a result, items were deleted or modified, leading to a 25-item test. The Appendix shows sample questions from the test. In the second pi- lot involving 371 senior undergraduate students, we assessed the factor structure of the test. An exploratory factor analysis with varimax rota- tion of the pilot test responses showed multiple factors with eigenvalues greater than one that collectively explained 58% of the variance but with no dominant first factor. This lack of distinct interpretable factors was expected based on arguments and past findings from situational judgment research (Chan, 2006; Chan & Shmitt, 2002; Lievens & Sackett, 2007) that suggest a construct measured by a situational judgment test can be viewed as “an aggregate composite ability consisting of multiple unitary or multidimensional competencies” (Chan, 2006, p. 478). This is consis- tent with our conceptualization of VT-SJ in this study. Given this, it was most meaningful to use an overall SJT score in our analysis.
Because past research has shown that situational judgment test scores can be highly correlated with cognitive ability and personality (for a re- view, see McDaniel et al., 2007), we also used data from the second pilot
HILL AND BARTOL 175
to assess the correlation between the VT-SJ test scores and these individ- ual differences. We found that participants’ test scores were significantly correlated with their scores on the Big Five personality traits (correla- tion varied between .12 for extraversion and .36 for agreeableness), but not significantly correlated with student participants’ grade point average, which we used as an indicator of cognitive ability. We conducted two additional pilots to provide some insight into the predictive validity of the VT-SJ test. The samples for the third and fourth pilots were, respectively, 158 senior undergraduate students and 70 MBA students with 1–3 years of professional work experience collaborating in teams that involved a signif- icant amount of virtual collaboration. Results from the third pilot showed that scores on the VT-SJ test predicted self-reported virtual collaboration beyond the influence of the Big Five personality variables (F(1,356) = 4.22, p < .05). In the fourth pilot, we found that the test scores signifi- cantly predicted virtual collaboration rated by peers in the team (B = .12, p < .001).
We proceeded to use the 25-item VT-SJ test in the current field study. The Cronbach’s alpha reliability coefficient for this test was 0.62, which is within the range acceptable for a test of this length and scoring ap- proach (i.e., dichotomous scoring: Kehoe, 1995; Lievens et al., 2005) and is comparable to other published situational judgment tests used in orga- nizational research (Lievens et al., 2005; Motowidlo, Dunnette, & Carter, 1990; Ployhart, Weekley, Holtz, & Kemp, 2003). The previously noted multidimensionality of situational judgment test items that require appli- cation of different areas of knowledge within the overall content domain can negatively impact measures of internal consistency; however, this test fell within acceptable ranges for this type of instrument. VT-SJ was mea- sured as the focal team member’s score on the situational judgment test we developed for this study (for a discussion and review of situational judgment tests, see McDaniel et al., 2001).
Team geographic dispersion. To measure team geographic dispersion we obtained information from the company indicating the locations for all team members. We used this data to compute a measure of team geographic dispersion for each team that included multiple dimensions of dispersion discussed in the literature that were relevant to the teams in this sample (O’Leary & Cummings, 2007; Schweitzer and Duxbury, 2010). These dimensions included the degree of separation or spatial distance between team members, the number of different countries, and the number of different work site locations represented in the team. We used Schweitzer and Duxbury’s (2010) operationalization of degree of separation, which involves selecting one team member’s location as a reference point and computing a distance index for all other team members based on whether they are located in the same city, continent, or hemisphere. Following the
176 PERSONNEL PSYCHOLOGY
approach taken in previous research (Hinds & Mortensen, 2005; Joshi et al., 2009), we conducted an exploratory factor analysis to examine the dimensionality of these constructs. The results showed that the three measures of dispersion loaded on to a single factor. Accordingly, we computed z-scores for each of these measures and combined them into a single index of geographic dispersion.
Virtual collaboration. At the individual level, we measured team mem- ber virtual collaboration with a 3-factor scale developed for this study that assesses the extent to which the focal team member engages in behaviors that support effective collaboration in geographically dispersed teamwork. In developing the measure, we consulted a review and synthesis of the dispersed team literature by Hertel et al. (2005) in which they developed a virtual teamwork competency model describing categories of behaviors that are particularly important for virtual collaboration. The two members of the research team also independently examined other major reviews (e.g., Axtell et al., 2004; Kirkman et al., 2012; Powell et al., 2004; Shin, 2004), comparing the categories of behaviors discussed in these reviews to those identified by Hertel et al. (2005) in order to identify refinements to the categories included in the competency model. The members of the research team then discussed the results of their independent exami- nations to come to agreement on the categories of behaviors underlying a team member’s virtual collaboration. These discussions noted consid- erable congruence across reviews with respect to important behaviors associated with virtual collaboration (e.g., Axtell et al., 2004; Kirkman et al., 2012; Powell et al., 2004; Shin, 2004).
After coming to agreement on the major categories of virtual collabo- ration behaviors, we developed items to reflect behaviors in each category based on descriptions of behaviors identified in Hertel et al.’s (2005) model and the other reviews. We also consulted the original articles identified in the reviews, as needed, for additional input into the wording of the items. We modified, combined, and dropped items as we developed them in order to reduce overlap and redundancy. This resulted in a reduced set of items categorized into the following major categories of behaviors: effective use of technology for virtual communication (e.g., Cramton, 2001; Goodhue & Thompson, 1995; Hinds & Weisband, 2003); supportive and responsive virtual interactions (interactions that facilitate task-based trust and effec- tive team coordination: Jarvenpaa & Leidner, 1999); and collaborating virtually across boundaries in the team (interactions aimed at leveraging team member differences and avoiding dysfunctional conflict: Cramton, Orvis, & Wilson, 2007; Hinds & Bailey, 2003, Hinds & Mortensen, 2005).
Prior to using the measure in the current field study, we included the measure in the first two pilot studies mentioned earlier to validate the overall reliability of the scales, examine the measure’s factor structure
HILL AND BARTOL 177
using exploratory factor analysis, and make modifications to the scale items. The scale used in the current field study consisted of three factors with 10 items (see Appendix). The first factor, communicating virtually using technology, had four items (e.g., “Uses technology effectively to communicate with team members”). The second and third factors had three items each: responsive virtual interactions (e.g., “Keeps team mem- bers informed of progress and issues”) and collaborating virtually across boundaries (e.g., “Is open to differences in ideas and approaches to the task among members of the team”). Peers indicated the extent to which each item described the focal team member using a scale of 1 = does not describe the team member at all to 7 = describes the team member extremely well.
CFA using data from this study on the virtual collaboration mea- sure showed acceptable fit for a model with three first-order factors (the three virtual collaboration dimensions) and one second-order factor (χ 2 = 287.20, df = 32, p < .001; non-normed fit index (NNFI) = .93, comparative fit index (CFI) = .95, standardized root mean square residual (SRMR) = .03). This model showed significantly better fit than a model in which all the behaviors were loaded on one factor (χ 2 = 403.25, df = 36, p < .001; NNFI = .91, CFI = .93, SRMR = .04). These CFA results, along with the high correlations between the virtual collaboration dimensions (r between .87 and .90), support the idea that these categories of behavior are distinct but collectively reflect a construct that describes virtual collaboration. We computed virtual collaboration for each focal team member by aggregating the mean ratings received from the peers in the team who provided an assessment of that focal team member. Both the rwg as well as the two intraclasss coefficients, ICC(1) and ICC(2), provided evidence of an acceptable level of agreement [median rwg = .96, ICC(1) = .09 (F = 1.29; p < .05), ICC(2) = .23] to justify aggregating peer scores into an overall mean score for the focal individual. We did not anticipate large ICC(2) values because the ICC(2) value is a function of the number of peer scores and the ICC(1) value (Bliese, 2000). In this case, the average number of peer assessors used to calculate the team member virtual collaboration was 2.83. Low ICC(2) values suggest that it may be difficult to uncover emergent relationships using group means (Bliese, 2000). However, such a circumstance should not preclude aggregation if it is warranted by theory and substantiated by high rwg figures (Chen and Bliese, 2002; Gelfand, Leslie, Keller, & de Dreu, 2012; Liao et al., 2009). At the team level, we were interested in the aggregated virtual collabo- ration of team members and so used the