+ - 0:00:00
Notes for current slide
Notes for next slide

Multilevel Modeling

Introduction and Recent Advances for Behavioral Research


Mark Lai

2018/12/04

1 / 58

Different Names

Hierarchical linear model (HLM; Raudenbush and Bryk, 2002)

Mixed/Mixed-effects model (Littell, Milliken, Stroup, and Wolfinger, 1996)

Random coefficient model (de Leeuw and Kreft, 1986)

Variance component model (Aitkin and Longford, 1986)

2 / 58

Roadmap

What are multilevel data?

Why use MLM?

  • Avoid underestimated SE
  • Cluster-specific (or person-specific) regression lines
  • Avoid Ecological Fallacy

Recent advances

3 / 58

What Are Multilevel Data?

4 / 58

Multilevel Data

Nested Data

  • Students in classrooms/schools
  • Siblings in families
  • Clients in therapy groups/therapists/clinics
  • Employees in organizations in countries

5 / 58

Multilevel Data

  • Repeated measures in individuals

Network Graph

boxes_and_circles A A 1 1 A->1 2 2 A->2 3 3 A->3 B B 4 4 B->4 5 5 B->5 C C 6 6 C->6 7 7 C->7 8 8 C->8 9 9 C->9 D D 10 10 D->10 11 11 D->11
6 / 58

Talk about level 1 and level 2

7 / 58

Applications of MLM

Psychotherapy

  • Heterogeneity of treatment effectiveness across therapists

Educational research

  • Teacher expectations on students' performance

Organizational research

  • Job strain and ambulatory blood pressure
8 / 58

Applications of MLM (cont'd)

Cross-national/neighborhood research

  • Sociopolitical influence on psychological processes (e.g., age and generalized trust)
  • Post-materialism, locus of control, and concern for global warming

Longitudinal analysis/repeated measures

  • Aging, self-esteem, and stress appraisal
9 / 58

Example Data

Sample simulated data on students' popularity

(Hox, Moerbeek, and Van de Schoot, 2018)

  • 2000 pupils (level 1) in 100 classrooms (level 2)

Main outcome:

  • popular: popularity teacher evaluation (0 to 10)
10 / 58
 
11 / 58

Intraclass Correlation

ICC = ave r between 2 obs (students) in same cluster (class)

12 / 58

Why Use MLM?

  • Avoid underestimated SE
  • Cluster-specific (or person-specific) regression lines
  • Avoid Ecological Fallacy
13 / 58

Dependent (Correlated) Observations

With clustered data, an assumption of OLS regression is violated

One score informs another score in the same cluster

Overlap: reduces effective information (\(N_\text{eff}\)) in data

14 / 58

Two students in the same class give less than two pieces of information

Consequences

Assuming independent obs, OLS understates the uncertainty in the estimates

  • SE too small; CI too narrow

$${\uparrow}\, t = \frac{\hat\beta}{\mathit{SE}(\hat \beta)\, \downarrow}$$

15 / 58

Comparing OLS with MLM

path1 texp texp popular popular texp->popular
OLS MLM
(Intercept) 4.205 (0.071) 4.197 (0.186)
texp 0.061 (0.005) 0.062 (0.012)
Num. obs. 2000 2000
Num. groups: class 100
16 / 58

OLS:
95% CI [0.052, 0.070].

MLM:
95% CI [0.038, 0.085].

17 / 58

Type I Error Inflation

Depends on design effect: 1 + (cluster size - 1) × ICC

18 / 58

Lai and Kwok (2015): MLM needed when design effect > 1.1

For the popularity data, design effect
= 1 + (20 - 1) × .365 = 7.934

\(N_\text{eff}\) reduces by almost 8 times: 2000 → 252

19 / 58

Why Use MLM?

  • Avoid underestimated SE
  • Cluster-specific (or person-specific) regression lines
  • Avoid Ecological Fallacy
20 / 58

Random Coefficient Model

Lv-1 predictor: not just problem on SE, OLS also ignore potential heterogeneity in regression lines

Consider extrav --> popular (with extrav mean centered)

path1 extravc extravc popular popular extravc->popular
21 / 58

OLS With All Data

22 / 58

Think About Just One Classroom

$$\texttt{popular}_i = \beta_0 + \beta_1 \texttt{extrav}_i + e_i$$

23 / 58

Think About Just One Classroom

$$\texttt{popular}_{i\color{red}{1}} = \beta_{0\color{red}{1}} + \beta_{1\color{red}{1}} \texttt{extrav}_{i\color{red}{1}} + e_{i\color{red}{1}}$$

24 / 58

Think About Classroom 35

$$\texttt{popular}_{i\color{blue}{35}} = \beta_{0\color{blue}{35}} + \beta_{1\color{blue}{35}} \texttt{extrav}_{i\color{blue}{35}} + e_{i\color{blue}{35}}$$

25 / 58

Classroom 14

$$\texttt{popular}_{i\color{green}{14}} = \beta_{0\color{green}{14}} + \beta_{1\color{green}{14}} \texttt{extrav}_{i\color{green}{14}} + e_{i\color{green}{14}}$$

26 / 58

MLM: efficiently get cluster-specific regression lines

$$\texttt{popular}_{i\color{purple}{j}} = \beta_{0\color{purple}{j}} + \beta_{1\color{purple}{j}} \texttt{extrav}_{i\color{purple}{j}} + e_{i\color{purple}{j}}$$

27 / 58

With heterogeneity in slopes, OLS gives underestimated SE (Lai and Kwok, 2015)

Detecting and explaining heterogeneity in slopes (i.e., cross-level interaction)

28 / 58

Growth Curve Analysis

Individual as "cluster"

29 / 58

Why Use MLM?

  • Avoid underestimated SE
  • Cluster-specific (or person-specific) regression lines
  • Avoid Ecological Fallacy
30 / 58

Ecological Fallacy

Association between two variables can be different across levels

path1 stu_ach Student academic achievement stu_asc Student academic self-concept stu_ach->stu_asc +ve
31 / 58

students' self-perception in academics

Ecological Fallacy

Association between two variables can be different across levels

path1 stu_ach Student academic achievement sch_ach School average academic achievement stu_ach->sch_ach stu_asc Student academic self-concept stu_ach->stu_asc +ve
32 / 58

Ecological Fallacy

Association between two variables can be different across levels

path1 stu_ach Student academic achievement sch_ach School average academic achievement stu_ach->sch_ach stu_asc Student academic self-concept stu_ach->stu_asc +ve sch_ach->stu_asc -ve
33 / 58

Big Fish Small Fond Effect

Ignoring clustering

34 / 58

Big Fish Small Fond Effect

+ve at student level

35 / 58

Big Fish Small Fond Effect

-ve contextual effect in a more competitive school

36 / 58

Why Use MLM?

  • Avoid underestimated SE
  • Cluster-specific (or person-specific) regression lines
  • Avoid Ecological Fallacy
37 / 58

Recent Advances

38 / 58

Other Forms of Clustering

Three-level

boxes_and_circles AB Therapist 1 A Therapy group A AB->A B Therapy group B AB->B CD Therapist 2 C Therapy group C CD->C D Therapy group D CD->D 1 1 A->1 2 2 A->2 3 3 A->3 4 4 B->4 5 5 B->5 6 6 C->6 7 7 C->7 8 8 C->8 9 9 C->9 10 10 D->10 11 11 D->11
39 / 58

Other Forms of Clustering

Cross-classification

boxes_and_circles n1 Neighborhood 1 n2 Neighborhood 2 n3 Neighborhood 3 A School A 1 1 A->1 2 2 A->2 3 3 A->3 B School B 4 4 B->4 5 5 B->5 C School C 6 6 C->6 7 7 C->7 8 8 C->8 9 9 C->9 D School D 10 10 D->10 11 11 D->11 1->n1 2->n2 3->n2 4->n1 5->n1 6->n2 7->n2 8->n3 9->n3 10->n3 11->n3

Partial Nesting

boxes_and_circles A Therapy group A 1 1 A->1 2 2 A->2 3 3 A->3 B Therapy group B 4 4 B->4 5 5 B->5 C Therapy group C 6 6 C->6 7 7 C->7 D Individual therapy 8 8 9 9 10 10 11 11
40 / 58

Effect Size for Multilevel Trials

Effect size (e.g., Cohen's d) required by many reporting standards

  • Little guidance on how to compute them
41 / 58

Effect Size for Multilevel Trials

Two-level trials: Hedges (2007)

Lai and Kwok (2014): (partially) cross-classified

Lai and Kwok (2016): partially nested

Rights and Sterba (2018): Defining \(R^2\) for MLM

42 / 58

Stice, Shaw, Burton, and Wade (2006)

Eating disorder prevention
Outcome: Thin-ideal internalization (TII)

43 / 58
boxes_and_circles A Treatment group A 1 1 A->1 2 2 A->2 3 3 A->3 B Treatment group B 4 4 B->4 5 5 B->5 C Treatment group C 6 6 C->6 7 7 C->7 D Control 8 8 9 9 10 10 11 11
Dissonance-based (Treatment) Expressive writing (Control)
\(N^T\) = 114 females \(N^C\) = 126 females
17 groups
\(n\) = 6 to 10
ICC = .08, \(\mathit{deff}\) = 1.5
44 / 58

\(\hat \beta_\text{TREAT}\) = -0.44, SE = 0.09[1]

How many SDs does that correspond to?

[1] OLS underestimates SE by 25%; falsely assuming full clustering underestimates SE by 15%.

45 / 58

Lai and Kwok (2016)

$$\hat \delta = \frac{\hat \beta_\text{TREAT}}{\hat \sigma^2_\text{person}}$$

$$V(\hat \delta) = \frac{V(\hat \beta_\text{TREAT})}{\hat \sigma^2_\text{person}} + \frac{\hat \delta^2 V(\hat \sigma^2_\text{person})}{4 (\hat \sigma^2_\text{person})^2}$$

For Stice, Shaw, Burton, et al. (2006), \(\hat \delta\) = -0.98, 95% CI = [-1.4, -0.6]

46 / 58

Multilevel Bootstrapping

Useful for

  • CI for effect size
  • testing multilevel mediation

bootmlm R package (Lai, 2018): currently implements 6 flavors of bootstrapping and 5 types of CI estimates

47 / 58
path1 free School poverty smorale School morale free->smorale late Student tardiness free->late smorale->late

48 / 58

Accounting for Survey Design

Lai, Kwok, Hsiao, and Cao (2018)

  • Apply finite population correction for cross-cultural research to obtain more accurate SE (and improved power)

Wen & Lai (in preparation)

  • A multilevel Bayesian semi-parametric bootstrap procedure to handle sampling weights with unequal probability samples
49 / 58

(Lai, Kwok, Hsiao, et al., 2018, Figure 5)

50 / 58

MLM of Non-Normal Outcomes

Ordinal? Counts? Zero-inflated? Proportions? Response time? Survival?

These are made easy with Bayesian estimation and the brms package (Bürkner, 2018).

51 / 58

Final Remarks

Reasons for MLM: correct SE, model heterogeneity, avoid ecological fallacy

Methodological research: translate methods in single-level research + methods for new RQ

More truly multilevel theories

52 / 58

Bibliography

Aitkin, M. and N. Longford (1986). "Statistical modelling issues in school effectiveness studies". In: Journal of the Royal Statistical Society. Series A (General) 149, pp. 1-43. DOI: 10.2307/2981882.

Bürkner, P. (2018). "Advanced Bayesian Multilevel Modeling with the R Package brms". In: The R Journal 10.1, pp. 395-411.

Hedges, L. V. (2007). "Effect sizes in cluster-randomized designs". In: Journal of Educational and Behavioral Statistics 32, pp. 341-370. DOI: 10.3102/1076998606298043.

Hox, J. J, M. Moerbeek and R. Van de Schoot (2018). Multilevel analysis: Techniques and applications. 3rd ed. New York, NY: Routledge.

53 / 58

Lai, M. H. C. (2018). marklhc/bootmlm: bootmlm: bootstrap resampling for multilevel models. DOI: 10.5281/zenodo.1879127.

Lai, M. H. C. and O. Kwok (2014). "Standardized mean differences in two-level cross-classified random effects models". In: Journal of Educational and Behavioral Statistics 39.4, pp. 282-302. DOI: 10.3102/1076998614532950.

Lai, M. H. C. and O. Kwok (2015). "Examining the rule of thumb of not using multilevel modeling: The “design effect smaller than two” rule". En. In: The Journal of Experimental Education 83.3, pp. 423-438. DOI: 10.1080/00220973.2014.907229.

Lai, M. H. C. and O. Kwok (2016). "Estimating standardized effect sizes for two- and three-level partially nested data". In: Multivariate Behavioral Research, pp. 1-17. DOI: 10.1080/00273171.2016.1231606.

Lai, M. H. C, O. Kwok, Y. Hsiao, et al. (2018). "Finite population correction for two-level hierarchical linear models.". In: Psychological Methods 23.1, pp. 94-112. DOI: 10.1037/met0000137.

54 / 58

Leeuw, J. de and I. Kreft (1986). "Random coefficient models for multilevel analysis". In: Journal of Educational Statistics 11, pp. 57-85. DOI: 10.2307/1164848.

Littell, R. C, G. A. Milliken, W. W. Stroup, et al. (1996). SAS System for mixed models. Cary, NC: SAS.

Raudenbush, S. W. and A. S. Bryk (2002). Hierarchical linear models: Applications and data analysis methods. 2nd ed. Thousand Oaks, CA: Sage.

Rights, J. D. and S. K. Sterba (2018). "Quantifying explained variance in multilevel models: An integrative framework for defining R-squared measures.". In: Psychological Methods. ISSN: 1939-1463. DOI: 10.1037/met0000184.

Stice, E, H. Shaw, E. Burton, et al. (2006). "Dissonance and healthy weight eating disorder prevention programs: A randomized efficacy trial.". In: Journal of Consulting and Clinical Psychology 74, pp. 263-275. DOI: 10.1037/0022-006X.74.2.263.

55 / 58

Thanks!

Slides created via the R package xaringan.

56 / 58

Fixed Effect Estimates

(intercept) 5.031 (0.097)
extravc 0.493 (0.025)
Var(intercept) 0.892
Var(slope) 0.026
Cov(int, slope) -0.134
Var(residual) 0.895

Average slope of extrav = 0.493, 95% CI [0.442, 0.543].

57 / 58

Random Effect Variance Estimates

(intercept) 5.031 (0.097)
extravc 0.493 (0.025)
Var(intercept) 0.892
Var(slope) 0.026
Cov(int, slope) -0.134
Var(residual) 0.895

Variance of slopes = 0.026 (SD = 0.161)

  • For a majority of schools (within +/- 1 SD), slope expected in the range [0.332, 0.654].
58 / 58

for linear relations, only need to know the intercept and the slope;

Different Names

Hierarchical linear model (HLM; Raudenbush and Bryk, 2002)

Mixed/Mixed-effects model (Littell, Milliken, Stroup, and Wolfinger, 1996)

Random coefficient model (de Leeuw and Kreft, 1986)

Variance component model (Aitkin and Longford, 1986)

2 / 58
Paused

Help

Keyboard shortcuts

, , Pg Up, k Go to previous slide
, , Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow