The Parasocial Relationships in Social Media Survey: What is it and why did I create it?

September 15, 2024 by Jonah Hall

Author: Austin Boyd, Ph.D.

Parasocial Rela​​tionships and Their Measures 

Coined by Horton and Wohl (1956), parasocial ​relationships are a ​type of relationship that is experienced between a spectator and a performer’s persona. Due to the nature of the interaction, these relationships are one-sided and cannot be reciprocated ​​by the performer with which they are made. At the time, television was the most effective medium through which parasocial relationships could be developed (Horton & Strauss, 1957; Horton & Wohl, 1956). However, as time and technology have progressed, the mediums for parasocial relationships to occur have expanded beyond television to include radio, literature, sports, politics, and social media, such as Facebook, Twitter, and, of course, YouTube.  

Over the past 65+ years, hundreds of articles have been published with different scales created to measure parasocial phenomena in a variety of different contexts. W​​hile many different scales exist, they are not interchangeable across contexts,​​ and none have been validated to measure parasocial relationships in a social media context. Many of the scales were developed for specific media contexts, and because of this, they do not lend well to assessing parasocial phenomena in other situations and other forms of media without modification.​ ​Using a measure that has not been validated, and is unsuitable for a population, may compromise the results, even if it was found to be valid in a different context (Stewart et al., 2012). Furthermore, research (e.g., Dibble et al., 2016; Schramm & Hartmann, 2008) has started to question the validity of these instruments. An assertion made by Dibble et al. (2016) states that most parasocial interaction scales have not undergone adequate tests of validation. 

The Parasocial Relationships in Social Media (PRISIM) Survey 

For my dissertation, I developed and began validating the scores of the Parasocial Relationships in Social Media (PRISM) Survey to measure the parasocial relationships that people develop with influences and other online celebrities through social media (Boyd et al., 2022; Boyd et al. 2024). The survey contains 22 items that were based on three well​​ established parasocial surveys: The Audience-Persona Interaction Scale (Auter & Palmgreen, 2000), Parasocial Interaction Scale (Rubin et al., 1985), and Celebrity-Persona Parasocial Interaction Scale (Bocarnea & Brown, 2007). ​​Participants are asked to indicate their level of agreement with each of the items using a five-point Likert scale.  

The 22 items comprise four constructs: Interest In, Knowledge Of, Identification With, and Interaction With. The first factor, ​Interest In​, contains seven items covering the level of concern for, perceived attractiveness of, and devotion to the celebrity and the content that they create. The second factor, ​Knowledge Of​, contains five items that deal specifically with the participants’ knowledge of the celebrity and desire to learn more about them. While similar to the Interest In construct, the items in this construct address the ​​participants’ curiosity and fascination with the celebrity, rather than their attachment to them. The third factor, ​Identification With​, includes six items addressing the perceived similarities, such as sharing qualities and opinions, between the celebrity and the participant. Finally, the fourth factor, ​Interaction With​, is four items which covers the social aspects involved with viewing the celebrity including the participants’ feelings of social and friendship connections with them. 

After creating the survey, we conducted a psychometric evaluation of the scale. This includes assessing the content, face, construct, convergent, and discriminant validity, as well as the internal consistency reliability and measurement invariance across different social media platforms. For full explanations of the methods and results used to validate the survey see Boyd et al. (2022) and Boyd et al. (2023). We have also created FAQ for those interested in using the survey, which can be found at https://austintboyd.github.io/prismsurvey/. Both articles and the PRISM survey have been published in open access journals to allow easy access for any researcher interested in conducting parasocial relationship research in the social media landscape. 

References 

Boyd, A. T., Morrow, J. A., & Rocconi, L. M. (2022). Development and Validation of the Parasocial Relationship in Social Media Survey. The Journal of Social Media in Society, 11(2), 192-208. 

Boyd, A. T., Rocconi, L. M., & Morrow, J. A. (2024). Construct Validation and Measurement Invariance of the Parasocial Relationships in Social Media Survey. PLoS ONE, 19(3). 

Dibble, J. L., Hartmann, T., & Rosaen, S. F. (2016). Parasocial interaction and parasocial relationship: Conceptual clarification and a critical assessment of measures. Human Communication Research, 42(1), 21–44. https://doi.org/10.1111/hcre.12063  

Horton, D., & Strauss, A. (1957). Interaction in Audience-Participation Shows. American Journal of Sociology, 62(6), 579–587. doi: 10.1086/222106 

Horton, D., & Wohl, R. R. (1956). Mass Communication and Para-Social Interaction. Psychiatry, 19(3), 215–229. doi: 10.1080/00332747.1956.11023049 

Schramm, H., & Hartmann, T. (2008). The psi-process scales. A new measure to assess the intensity and breadth of parasocial processes. Communications, 33(4). https://doi.org/10.1515/comm.2008.025  

Stewart, A. L., Thrasher, A. D., Goldberg, J., & Shea, J. A. (2012). A framework for understanding modifications to measures for diverse populations. Journal of Aging and Health, 24(6), 992–1017. https://doi.org/10.1177/0898264312440321  

mlmhelper: An R helper package for estimating multilevel models

September 1, 2024 by Jonah Hall

Author: Louis Rocconi, Ph.D.

Multilevel modeling, also known as linear mixed modeling or hierarchical linear modeling, is a statistical technique used to analyze nested data (e.g., students in schools) or longitudinal data. Both nested and longitudinal data often result in correlated observations, violating the assumption of independent observations. This issue is common in educational, social, health, and behavioral research. Multilevel modeling addresses this by modeling and accounting for the variability at each level, leading to more accurate estimates and inferences. 

The inspiration for developing mlmhelpr came from my experience teaching a multilevel modeling course. Throughout the semester, I found myself repeatedly writing custom functions to help students perform various tasks mentioned in our readings. Additionally, students often expressed frustration that lme4, the primary R package for estimating multilevel models, did not provide all the necessary information required by our textbooks and readings. After the semester ended, Anthony and I discussed the need to consolidate these functions into a single R package, making them accessible to everyone. mlmhelpr offers tests and statistics from many popular multilevel modeling textbooks such as Raudenbush and Bryk (2002), Hox et al. (2018), and Snijders and Bosker (2012), and like every other R package, it is free to use! 

The following is a list of package functions and descriptions.  

boot_se 

This function computes bootstrap standard errors and confidence intervals for fixed effects. This function is mainly a wrapper for lme4::bootMer function with the addition of confidence intervals and z-tests for fixed effects. This function can be useful in instances where robust_se does not work, such as with nonlinear models (e.g., glmer models). 

center 

This function refits a model using grand-mean centering, group-mean/within cluster centering (if a grouping variable is specified), or centering at a user-specified value. For additional information on centering variables in multilevel models, see Enders and Tofighi (2007). 

design_effect 

This function calculates the design effect, which quantifies the degree to which a sample deviates from a simple random sample. In the multilevel modeling context, this can be used to determine whether clustering will bias standard errors and whether the assumption of independence is held. 

hausman 

This function performs a Hausman test to test for differences between random- and fixed-effects models. This test determines whether there are significant differences between fixed-effect and random-effect models with similar specifications. If the test statistic is not statistically significant, a random effects model (i.e. a multilevel model) may be more suitable (i.e., efficient). The Hausman test is based on Fox (2016, p. 732, footnote 46). I consider this function experimental and would interpret the results with caution. 

icc 

This function calculates the intraclass correlation. The ICC represents the proportion of group-level variance to total variance. The ICC can be calculated for two or more levels in random-intercept models. For models with random slopes, it is advised to interpret results with caution. According to Kreft and De Leeuw (1998, p. 63), “The concept of intra-class correlation is based on a model with a random intercept only. No unique intra-class correlation can be calculated when a random slope is present in the model.” However, Snijders and Bosker (2012) offer a calculation to derive this value (equation 7.9), and their approach is implemented. For logistic models, the estimation method follows Hox et al. (2018, p. 107). For a discussion of different methods for estimating the intraclass correlation for binary responses, see Wu et al. (2012). 

ncv_tests 

This function computes three different non-constant variance tests: (1) the H test as discussed in Raudenbush and Bryk (2002, pp. 263-265) and Snijders and Bosker (2012, p. 159-160), (2) an approximate Levene’s test discussed by Hox et al. (2018, p. 238), and (3) a variation of the Breusch-Pagan test. The H test computes a standardized measure of dispersion for each level-2 group and detects heteroscedasticity in the form of between-group differences in the level-one residual variances. Levene’s test computes a one-way analysis of variance of the level-2 grouping variable on the squared residuals of the model. This test examines whether the variance of the residuals is the same in all groups. The Breusch-Pagan test regresses the squared residuals on the fitted model. A likelihood ratio test is used to compare this model with a null model that regresses the squared residuals on an empty model with the same random effects. This test examines whether the variance of the residuals depends on the predictor variables. 

plausible_values 

This function computes the plausible value range for random effects. The plausible values range is useful for gauging the magnitude of variation around fixed effects. For more information, see Raudenbush and Bryk (2002, p. 71) and Hoffman (2015, p. 166). 

r2_cor 

This function calculates the squared correlation between the observed and predicted values. This pseudo R-squared is similar to the R-squared used in OLS regression. It indicates the amount of variation in the outcome that is explained by the model (Peugh, 2010; Singer & Willett, 2003, p. 36). For additional pseudo-R2 measures, see the r2glmm and performance packages. 

r2_pve 

This function computes the proportion of variance explained for each random effect level in the model (i.e., level-1, level-2) as discussed by Raudenbush & Bryk (2002, p. 79). For additional pseudo-R2 measures, see the r2glmm and performance packages. 

reliability 

This function computes reliability coefficients for random effects according to Raudenbush and Bryk (2002) and Snijders and Bosker (2012). The reliability coefficient indicates how much of the variance in the random effect is due to true differences between clusters rather than random noise. The empirical Bayes estimator for the random effect combines the cluster mean and the grand mean, with the weight determined by the reliability coefficient. A reliability close to 1 puts more weight on the cluster mean while a reliability close to 0 puts more weight on the grand mean. 

robust_se 

This function computes robust standard errors for linear models. It is a wrapper function for the cluster-robust standard errors from the clubSandwich package that includes confidence intervals. See the clubSandwich package for additional information and mlmhelpr::boot_se for an alternative. 

taucov 

This function calculates the covariance between random intercepts and slopes. It is used to quickly get the covariance and correlation between intercepts and slopes. By default, lme4 only displays the correlation. 

As of August 26, 2024, the package has been downloaded 5,062 times! If you use R to estimate multilevel models, give mlmhelpr a try, and let me know if you find any errors or mistakes. I hope you find it helpful. If you are interested in creating your own R package, check out the excellent R Package development book by Wickham and Bryan: https://r-pkgs.org/.  

Happy modeling!  

Resources and References 

Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121-138.  

Fox, J. (2016). Applied regression analysis and generalized linear models (3rd ed.). SAGE Publications. 

Hoffman, L. (2015). Longitudinal analysis: Modeling within-person fluctuation and change. Routledge. 

Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). Routledge. 

Kreft, I. G. G., & De Leeuw, J. (1998). Introducing multilevel modeling. SAGE Publications. 

Peugh, J. L. (2010). A practical guide to multilevel modeling. Journal of School Psychology, 48(1), 85-112.  

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE Publications. 

Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford University Press. 

Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). SAGE Publications. 

Wu, S., Crespi, C. M., & Wong, W. K. (2012). Comparison of methods for estimating the intraclass correlation coefficient for binary responses in cancer prevention cluster randomized trials. Contemporary Clinical Trials, 33(5), 869. 

Organizing your Evaluation Data: The Importance of Having a Comprehensive Data Codebook

August 16, 2024 by Jonah Hall

By J.A. Morrow

Why a Data Codebook? 

One of my pet peeves is a disorganized project and inconsistency in how data are organized. For every project, whether it is an evaluation research or assessment project, I start developing a data codebook before I even begin data collection. When I take on a new project from an evaluation or assessment client, I first ask for their codebook or if they don’t have one then I create it for them. Why is this so important, you ask? Think of your codebook as your organizational tool and project history all rolled into one document. It contains everything about your project and greatly aids in getting everyone on your team organized and on the same page. Your clients (and your future self) will greatly appreciate this too!  

Your data codebook is a living document, it changes throughout the life of a project as you add new data, modify data, and make decisions throughout the course of the project. Not having a data codebook can lead to confusion and increase the chances of someone on your team making a mistake when analyzing data and disseminating information to your clients. Sadly, I have sat through presentations where a client points out a mistake or has a question about the data that can’t be answered by the evaluation team because they don’t have a record of what was done. Clients are never happy when this happens! 

What is in a Data Codebook? 

I usually include the following 9 things in my data codebooks: 

1. Name of the Evaluation Project 

2. Variable Names 

3. Variable Labels 

4. Value Labels 

5. Newly Created/Modified Variables (and how you created/modified these) 

6. Citations for Scales and Sources of Data for the Project 

7. Reliability of any Composite Items 

8. List of Datasets and Sample Size for Each 

9. Project Diary/Notebook 

I typically put the first 7 in one table, which I create in Microsoft Word. You can also create your codebook using Excel or any other analysis software package (e.g, SPSS, R). This first table provides details about all of the data for a project. As I make any changes to the datasets, I add any new variables that I create to this table and write up my decision making for any changes in the project diary/notebook section of my codebook. 

For the list of datasets and sample sizes I usually have that as a separate table at the end of my codebook. As I create a new dataset or project file I enter that information in this section of the codebook. I also include a brief description of what is contained in the new data file. I always organize this table by the most recent files first. 

Lastly, I include an extensive project diary/notebook at part of my codebook. For some projects these can be very long and have many team members adding to it so I typically will have this as a document link in the codebook. The document link takes team members to an external Google document where we all can write and edit information about what we are working on for the project and what decisions were made. I cannot overstate how important it is to have a detailed project diary/notebook for an evaluation project. It is especially useful as you are writing your reports for your client about what you did and why you did something in a particular way. Anytime I have a project meeting with my team or a meeting with my client I take notes in our project notebook. 

Additional Advice 

So, I hope I have provided some useful tips as you start the process of organizing your evaluation data. One last piece of advice….share this codebook with your client! At the end of a project, I give the codebook (minus the project notebook as that is internal to my team) and final datasets (sanitized at some level depending on the contract) to my client so they can continue to utilize the data for their program/organization. Empower your evaluation clients to better understand their data and how their data was processed! 

Resources 

12 Steps of Data Cleaning Handout:
https://www.dropbox.com/scl/fi/x2bf2t0q134p0cx4kvej0/TWELVE-STEPS-OF-DATA-CLEANING-BRIEF-HANDOUT-MORROW-2017.pdf?rlkey=lfrllz3zya83qzeny6ubwzvjj&dl=0 

https://datamgmtinedresearch.com/document

https://dss.princeton.edu/online_help/analysis/codebook.htm

https://ies.ed.gov/ncee/rel/regions/central/pdf/CE5.3.2-Guidelines-for-a-Codebook.pdf

https://libguides.library.kent.edu/SPSS/Codebooks

https://web.pdx.edu/~cgrd/codebk.htm

https://www.datafiles.samhsa.gov/get-help/codebooks/what-codebook

https://www.icpsr.umich.edu/web/ICPSR/cms/1983

https://www.medicine.mcgill.ca/epidemiology/joseph/pbelisle/CodebookCookbook/CodebookCookbook.pdf

https://www.slideshare.net/sl

The Conventional and Unconventional Places I’ve Found Research Ideas

August 1, 2024 by Jonah Hall

By Dr. Austin Boyd

Research is the backbone of the academic world. It provides us with a ​​​​better understanding of the world around us and allows knowledge to be passed on and built upon by future generations of researchers. Research may be conducted as​ a​ class project, larger end of course capstone or dissertation requirement, or as ​a ​regular part of many careers in and outside of academia. But where does one come up with a research idea? For some, finding inspiration for research projects is easy, resulting in a laundry list of ideas to pick and choose from. For others, coming up with even a single research idea might take longer than the research itself. Compound this with the fact that some fields have existed for decades, or even centuries, and it might feel as though there is nothing left to research. However, as daunting as it may seem, there are always more questions to be asked, you just need to ​​keep an open mind.

Project 1: A Student with a Question

My first research idea came about as conventionally as ​they​ come. I was a student with a question, and with the guidance of a professor, we came up with a research idea and then pursued it. Once upon a time, I took a statistics course on Item Response Theory (IRT). While sitting in class one day, we were discussing the underlying assumptions of IRT presented in Embretson and Reise’s Psychometric Methods: Item response theory for psychologists (2000). After class, I ​approached​ my professor with a question: “How does skewness impact measurement invariance?” Little did I know, this was a question she had always wondered herself, but never had the time to pursue. Over the next few weeks in office hours, we discussed ideas on how to address this question, and before long, she told me that she could provide me with data if I would be interested in exploring the topic further. Over the course of the next three years, she and I worked to test the robustness of this assumption, ​and​ ended up presenting our findings at two conferences and published them in the Journal of Applied Measurement (Boyd et al., ​​2020).  

Project 2: Friends Talking About Movies

Project 3: Watching YouTube to Avoid Schoolwork

My third research idea was born out of avoiding schoolwork. In one of my graduate courses, we had to develop and design an original survey on a topic of our choosing. There were many steps to this project, ​the first of which was ​​imply to propose an idea for the survey. Instead of doing that, I was watching people play video games on YouTube. After a while, I started wondering what makes online celebrities and influencers so popular, and after a quick Google search, I learned about a concept called parasocial relationships. These are the one-sided relationship​s​ that a viewer makes with a performer (Horton & Wohl, 1956). I kept digging into the topic and learned that people have been researching parasocial relationships and interactions for over 60 years, long before YouTube, or even the internet, existed. ​S​everal surveys had already been developed for ​understanding​​     ​ parasocial relationships and interactions with television personalities, TV characters, fictional characters, and even political candidates, but none for social media based celebrities and influencers. I decided that this would be the topic of my survey. Over the course of the semester, I developed my survey and put together a proposal for how I would pilot and assess the reliability and validity of the survey. I could have walked away from it once the course was over, but I came back a year later once I realized that this survey could actually be the basis for a real research project. As a result, my one class project became the basis for my entire dissertation and yielded two publications on the development and validity of the Parasocial Relationship in Social Media (PRISM) Survey (Boyd et al., 2022; Boyd et al. 2023).​ 

Research ideas are everywhere, even when it seems like there is nothing left to explore. And when it feels this way there are ​​five places I suggest taking a look:  

1. Prior literature – Prior literature if full of research ideas. Many publications include a section on future research ideas in the discussion, some of which are never fully explored. This can be a great place to start with a new research interest. 

2. Old class projects – Returning to an idea after being away from it can provide a new outlook that sparks a research idea. Coming back to an old project with the new knowledge gained from working on others can be revitalizing. 

3. Other researchers – Whether they be professors or peers, other researchers can be a great sounding board for ideas. Their knowledge and experiences can provide different points of view that can help inspire new project ideas. Some might even share ideas that they don’t have the time or interest in pursuing. 

4. Personal hobbies and interests – It might seem weird, but even personal interests can lead to research ideas. Without my interest in Marvel movies and YouTube, neither of my projects would have existed. 

5. ​​​Friends and family – Even if they don’t understand your research, sometimes talking to friends and family about it can spark new ideas. Their lack of knowledge on the subject can bring up questions that you never even thought about.​     ​ 

References:

​​​Boyd, A. T., Rocconi, L. M., & Morrow, J. A. (2024). Construct validation and measurement invariance of the Parasocial Relationships in Social Media Survey. PLoS ONE. 

Boyd, A. T., Morrow, J. A., & Rocconi, L. M. (2022). Development and validation of the Parasocial Relationship in Social Media Survey. The Journal of Social Media in Society, 11(2), 192-208. Available online: https://www.thejsms.org/index.php/JSMS/article/view/1085 

Boyd, A. T., & Rocconi, L. M. (2021). Formatting data for one and two mode undirected social network analysis. Practical Assessment, Research & Evaluation, 26(24). Available online: https://scholarworks.umass.edu/pare/vol26/iss1/24/ 

Boyd, A. T., Schmidt, K. M., & Bergeman, C. S. (2020). You know what they say about when you assume: Testing the robustness of invariant comparisons. Journal of Applied Measurement, 21(2), 190-209. 

Embretson, S. E., & Reise, S. (2000). Psychometric methods: Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum. 

Horton, D., & Wohl, R. R. (1956). Mass communication and parasocial interaction. Psychiatry, 19(3), 215–229. doi: 10.1080/00332747.1956.11023049 

Finding Your People: The Importance of Mentorship and Networking Early On

July 1, 2024 by Jonah Hall

By Richard Amoako

Allow me to share a brief anecdote that underscores the power of networking and mentorship. During my first year of graduate school at Ensign Global College, Ghana, I attended a “Welcome Reception”, feeling quite lost and overwhelmed. However, I mustered up the courage to introduce myself to a continuing student, who not only offered valuable insights into navigating the program but also connected me with a faculty member whose research aligned with my interests. That single conversation opened doors for me, leading to a fruitful mentorship relationship and even collaborative research projects and publications.

The Importance of Mentorship

Finding a mentor can be a game-changer, especially at the early stages of your career or academic endeavors. According to a publication by the American Psychological Association (APA), mentorship plays a crucial role in professional development, providing guidance, support, and opportunities for growth (Calkins, 2023). A mentor can serve as a trusted guide, offering advice and sharing industry knowledge that can accelerate your professional growth. They can provide insights into navigating challenges, identifying opportunities, and making informed career decisions.

Having a diverse set of mentors can significantly enhance your academic and career journey by providing support across different areas of your development. For example: A faculty mentor can offer deep insights into your field, guide your research, and help you navigate academic challenges; professional mentors bring industry knowledge and can guide you on career opportunities beyond academia, while peer mentors offer mutual support and foster a sense of community within your cohort. Additionally, cross-disciplinary mentors can help you think outside the box and provide insights into interdisciplinary collaboration, enhancing the breadth of your research and professional network (Chandler, 2011).

One of the best places to look for potential mentors is within your institution or organization. Seek out experienced professionals who inspire you and whose career paths align with your aspirations. Professional associations and conferences can be excellent platforms for connecting with potential mentors from diverse backgrounds and institutions. Not everyone may feel comfortable or confident approaching potential mentors at such events. As an introverted PhD student, I understand the challenge of navigating the bustling environment of conferences and professional events. Crowded spaces and numerous distractions can make it difficult to connect with potential mentors.

At one of the first major conferences I attended this year (American Educational Research Association, AERA, 2024), I felt overwhelmed by the sheer number of participants and the constant buzz of activity. Determined to make the most of the opportunity, I signed up for a small roundtable discussion focused on my research area. The roundtable provided a quiet setting where I could comfortably share my ideas, ask questions, and have meaningful interactions with panelists and facilitators.

When reaching out to potential mentors, it’s essential to approach the relationship with humility, respect, and a genuine desire to learn. Whether it’s through informational interviews, mentorship programs, or faculty-student collaborations, fostering meaningful connections with mentors can provide invaluable insights and open doors to opportunities that may otherwise remain out of reach.

Networking Strategies

Attending conferences and events such as professional development workshops, career fairs, research symposiums and colloquiums, Special Interest Groups (SIGs) and Meetups are crucial aspects of building your professional network in the evaluation community. These gatherings offer invaluable opportunities to connect with peers, established professionals, and potential collaborators from across the field.

To make the most of these events, prepare an elevator pitch, actively engage in conversations, and follow up with new connections after the event. The American Evaluation Association’s (AEA) annual conference is a good example of a major event where evaluators from around the world gather to share their work, learn from each other, and expand their networks. I had a great time when I attended the AEA evaluation conference for the first time in 2023 in Indianapolis, Indiana. The conference not only provided me with a platform to showcase my research but also allowed me to engage with like-minded individuals who share my passion for evaluation. Whether it’s striking up conversations during coffee breaks or attending panel discussions, conferences offer ample opportunities to expand your network and gather insights from seasoned professionals and industrial experts.

Joining professional organizations, such as the AEA, the AERA, or the Association for Institutional Research (AIR), among others, can also open up a wealth of networking opportunities. These organizations often offer local chapter meetings, online forums, and special interest groups, allowing you to connect with like-minded individuals and stay up-to-date with the latest trends and developments in your field. You will find that they not only foster a sense of belonging, but also offer avenues for professional development and growth.

In today’s digital age, building connections online has become increasingly important (Virk, 2023). Platforms like LinkedIn and X (formally Twitter) provide a space to showcase your professional profile, connect with others in your field, and engage in discussions within relevant industry groups and communities. These digital platforms provide ongoing networking opportunities, allowing professionals to connect and engage with each other continuously, rather than being limited to specific events (Pew Research Center, 2021). For Instance, the AEA, like other professional associations, has an active presence on LinkedIn. This platform also has groups dedicated to fostering discussions and sharing resources among evaluators.

Making Connections and Building Relationships

Effective networking is not just about collecting business cards or adding connections on LinkedIn. It’s about building genuine, mutually beneficial relationships. This involves actively listening, asking thoughtful questions, and showing a genuine interest in others’ work and experiences. As noted by Janasz and Forret (2008), successful networking relies on cultivating strong interpersonal ties and fostering a sense of reciprocity.

When making new connections, look for opportunities to offer value and share your knowledge and expertise. This could involve collaborating on projects, co-authoring publications, or simply providing insightful feedback and support. By positioning yourself as a valuable resource, you increase the likelihood of fostering long-lasting, meaningful relationships within your professional or academic community.

Overcoming Challenges and Staying Motivated

For many, the idea of networking can be daunting, especially for those who identify as introverted or shy. However, it’s important to remember that networking is a skill that can be developed with practice and persistence. Start small, perhaps by attending a local meetup or joining an online community, and gradually build your confidence. As posited by de Janasz and Forret (2008), setting achievable goals, and celebrating small wins can help overcome the initial hesitation and discomfort associated with networking.

These are the strategies I have adopted over the past few years: Before attending a conference, I find it helpful to do some homework. I review the conference program and identify sessions, workshops, and speakers that align with my research interests. By pinpointing the key individuals I’d like to connect with, I can set manageable goals for the event. This preparation not only eases the anxiety of large crowds but also provides a clear roadmap for meaningful interactions. Moreover, I focus on smaller, more manageable interactions instead of trying to network in large groups. For instance, I look for opportunities to engage with speakers after their presentations. These moments often provide a quiet setting for a brief, yet impactful conversation. Additionally, attending smaller workshops or special interest group meetings can offer a more intimate environment, conducive to connecting with like-minded individuals.

Consistency is key when it comes to networking efforts. Building a strong network takes time and commitment, so celebrate small wins and progress along the way. This will help you stay motivated and focused on your long-term goals of establishing meaningful connections within the evaluation community.

Conclusion

As an early-career professional in the field of evaluation, surrounding yourself with a supportive network of mentors and peers can make a significant difference in your personal and professional growth. By actively seeking out mentorship opportunities, attending conferences and events, joining professional organizations, and building meaningful connections, you’ll not only expand your knowledge and skills but also gain invaluable insights and perspectives that can shape your career trajectory.

I urge you to embrace the power of mentorship and networking early on in your journey. Whether you’re attending conferences, joining professional organizations, or seeking out mentors, remember that building a support network is not just about furthering your career—it’s about finding your tribe, your people who will uplift and empower you every step of the way.

Remember, the journey of building a strong network starts with taking that first step. So, don’t hesitate – start exploring opportunities to connect with others in your field today. The relationships you forge now could open doors to exciting collaborations, rewarding mentorships, and a fulfilling career path.

I hope this blog post has inspired you to prioritize networking and mentorship as you navigate the early stages of your career. Wishing you all the best in your networking endeavors!

References:

Calkins, H (2023). How to navigate the dynamics of mentorship. Knowing the best ways to handle challenges and conflict is crucial to being a good mentor. https://www.apa.org/monitor/2023/01/dynamics-mentorship

Chandler, D. E. (2011). The Maven of Mentoring Speaks: Kathy E. Kram Reflects on Her Career and the Field. Journal of Management Inquiry, 20(1), 24-33. https://doi.org/10.1177/1056492610369937

de Janasz, S. C., & Forret, M. L. (2008). Learning The Art of Networking: A Critical Skill for Enhancing Social Capital and Career Success. Journal of Management Education, 32(5), 629-650. https://doi.org/10.1177/1052562907307637

Havard (2022). How to Give a Great Elevator Pitch (With Examples). https://careerservices.fas.harvard.edu/blog/2022/09/07/how-to-give-a-great-elevator-pitch-with-examples/

Janasz, S. & Forret, M. (2008). Learning The Art of Networking: A Critical Skill for Enhancing Social Capital and Career Success. Journal of Management Education, 32. 629-650. https://doi.org/10.1177/1052562907307637

Pew Research Center. (2021). Social media fact sheet. Retrieved from https://www.pewresearch.org/internet/wp-content/uploads/sites/9/2021/04/PI_2021.04.07_Social-Media-Use_FINAL.pdf

Virk, S (2023). Connecting in the digital age: Navigating technology and social media. VISUAL LIFE. https://rikithompson.ds.lib.uw.edu/visuallife/connecting-in-the-digital-age-navigating-technology-and-social-media/

Resources:

Vinnie Malcolm, The Mutual Benefits of Mentorship https://www.youtube.com/watch?v=2lCjjlLK2m8

Benefits of Mentorship https://www.youtube.com/watch?v=a4dD0Ch4T4I

Why is Networking Important? https://www.youtube.com/watch?v=T2I9odCTILA

Professional Networking 101 https://www.youtube.com/watch?v=Xt-VdqXhHZM

How to NETWORK for career & jobs | Networking tips for professionals https://www.youtube.com/watch?v=IO5Ht7yV_0A

To Evaluate, or to Be Evaluated? That is the Question.

June 15, 2024 by Jonah Hall

By M. Andrew Young

Evaluation practices are becoming increasingly utilized across many industries. The American Evaluation Association (AEA) lists general industries on its careers page (Consulting, Education/Teaching/Administration, Government/Civil Service, Healthcare/Health Services, Non-profit/Charity, Other) (Evaluation Jobs – American Evaluation Association, n.d.). A quick Google search also indicates numerous other business-related evaluation opportunities (What Industries Employ Evaluators – Google Search, 2024).  

Why do I bother stating the obvious? Evaluation is everywhere! 

Simple. As an emerging evaluator (this is a whole different discussion, but in short, I’m newer to the field, so I’m “emerging”), it is important to critically reflect upon what it means to be an evaluator as a professional identity. Medical doctors have “M.D.” or “D.O.” degrees, and after an initial licensing process, they have an ongoing licensure examination as well as continuing education and conduct requirements (FSMB | About Physician Licensure, 2024). Physical Engineers have similar requirements such as an initial licensure requirement and continuing education (Maintaining a License, 2024). Pharmacists also have to pass licensure examinations (one national exam, the NAPLEX, and one state-specific exam, the MPJE) and also have continuing education requirements (Pharmacist Licensing Requirements & Service | Harbor Compliance | www.Harborcompliance.Com, 2024). 

Why do they do this? Education helps, experience helps, but why do these few aforementioned professions require licensure and continuing education as part of their right to practice their profession?  

Professional identity is an important function of entrustment given to a profession, and I pose the question: Can licensure with a continuing education requirement support that trust given to evaluators? It may be time to consider to what extent credentialing would support entrustment by those affected by or participating in evaluation activities. 

In the “early days” of the AEA in the 1990’s, the subject of credentialing was broached, and there was such sharp dissent about how to handle this, that AEA pushed back addressing that until their formed their AEA Competency Task Force. In 2015, “The Task Force believed that without AEA agreement about what competencies were essential, it was premature to decide how these competencies would be measured and monitored. Efforts such as the viability and value of adopting a credentialing or assessment system can be the task of working groups that follow ours” (Tucker et al., 2020). 

AEA is in good company without a licensure or credentialing requirement as they follow the example of other major evaluation societies that do not also require or offer credentialing (to my knowledge, only the CES and JES offer this at this point in time) (Altschuld, 1999; Ayoo et al., 2020; Tucker et al., 2020). 

What does it mean to me? 

In a rather pragmatic sense, a credentialing requirement would add a barrier to entry that would protect the economy of evaluation. A continuing education requirement would help make sure that practitioners in evaluation are also keeping current, and a conduct policy would help ensure ethical practice of evaluation. All-in-all, it would hopefully maintain the quality of the profession. While I have not explored how pervasive “bad” evaluation practice is, the more people doing evaluation as it continues to grow as a practice could open the doors for inexperienced and unknowledgeable evaluators to practice. “What about people who are doing some evaluation work for employers but aren’t a ‘professional’ evaluator?” you may ask. Good question. I’ll answer: people who do evaluation work as a part of their private employment would not be required to be licensed or credentialed, but having a license or credential might give them leverage to advance their careers and get compensated commensurate with their abilities. One does not have to be licensed in a software language to use it in the context of employment, but a person with a credential (usually in the form of a certificate embedded in a degree program) in software languages can ask for more compensation because they have demonstrated competence and thereby their employer can give entrustment to them to perform the tasks they will be asked to complete. 

Credentialing has been touted to do much more to the profession than what I listed above, and for that, here is a cool resource to read on this:  

• Ayoo, S., Wilcox, Y., LaVelle, J. M., Podems, D., & Barrington, G. V. (2020). Grounding the 2018 AEA Evaluator Competencies in the Broader Context of Professionalization. New Directions for Evaluation, 2020(168), 13–30. https://doi.org/10.1002/ev.20440 

What’s the downside? 

Well, like anything, there can be negative implications to credentialing. First, a credentialing body must be formed; second, credentialing requirements must be developed, adopted, and implemented. Then there is the question of what to do with the evaluators already practicing in the field? Then the licensure examination must be maintained. The list goes on, and formalizing the credentialing of evaluators can get very expensive and become a very large endeavor. Pharmacy faced this change when the industry moved from a bachelor’s degree requirement to a PharmD program in 1997. Their solution was to allow BPharm and previously-licensed pharmacists to continue to practice, and the accrediting body allowed colleges of pharmacy to offer two-year “upgrades” from a BPharm to a PharmD program for pre-existing licensed pharmacists (Supapaan et al., 2019).  

Second, and just as important, how do we design and implement a credentialing process that is both equitable and sustainable? 

Conclusion: 

Harkening back to the Shakespearean title, I leave you with this: 

“To evaluate, or to be evaluated, that is the question: 

Whether ‘tis nobler in the mind to suffer the slings and arrows of a burdensome credentialing process, 

Or to take arms against the lack of professional identity, 

And by adopting a credentialing process, end them. 

To credential – to license. 

No more; and by credentialing process to say we end the confusion of ‘who am I?’ and the thousand questions of entrustment that our profession is heir to: ‘tis a consummation devoutly to be wish’d.” 

References: 

Altschuld, J. W. (1999). The Certification of Evaluators: Highlights from a Report Submitted to the Board of Directors of the American Evaluation Association. American Journal of Evaluation, 20(3), 481–493. https://doi.org/10.1177/109821409902000307 

Ayoo, S., Wilcox, Y., LaVelle, J. M., Podems, D., & Barrington, G. V. (2020). Grounding the 2018 AEA Evaluator Competencies in the Broader Context of Professionalization. New Directions for Evaluation, 2020(168), 13–30. https://doi.org/10.1002/ev.20440 

Clarke, P. A. (2009). Leadership, beyond project management. Industrial and Commercial Training, 41(4), 187–194. https://doi.org/10.1108/00197850910962760 

Evaluation Jobs—American Evaluation Association. (n.d.). Retrieved March 23, 2024, from https://careers.eval.org?site_id=22991 

FSMB | About Physician Licensure. (2024). https://www.fsmb.org/u.s.-medical-regulatory-trends-and-actions/guide-to-medical-regulation-in-the-united-states/about-physician-licensure/ 

Gill, S., Kuwahara, R., & Wilce, M. (2016). Through a Culturally Competent Lens: Why the Program Evaluation Standards Matter. Health Promotion Practice, 17(1), 5–8. https://doi.org/10.1177/1524839915616364 

Jarrett, J. B., Berenbrok, L. A., Goliak, K. L., Meyer, S. M., & Shaughnessy, A. F. (2018). Entrustable Professional Activities as a Novel Framework for Pharmacy Education. American Journal of Pharmaceutical Education, 82(5), 6256. https://doi.org/10.5688/ajpe6256 

Kumas-Tan, Z., Beagan, B., Loppie, C., MacLeod, A., & Frank, B. (2007). Measures of Cultural Competence: Examining Hidden Assumptions. Academic Medicine, 82(6). https://journals.lww.com/academicmedicine/fulltext/2007/06000/measures_of_cultural_competence__examining_hidden.5.aspx 

Liphadzi, M., Aigbavboa, C. O., & Thwala, W. D. (2017). A Theoretical Perspective on the Difference between Leadership and Management. Creative Construction Conference 2017, CCC 2017, 19-22 June 2017, Primosten, Croatia, 196, 478–482. https://doi.org/10.1016/j.proeng.2017.07.227 

Maintaining a License. (2024). National Society of Professional Engineers. https://www.nspe.org/resources/licensure/maintaining-license 

Pharmacist Licensing Requirements & Service | Harbor Compliance | www.harborcompliance.com. (2024). https://www.harborcompliance.com/pharmacist-license 

SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. New Directions for Evaluation, 2004(102), 5–19. https://doi.org/10.1002/ev.112 

Supapaan, T., Low, B. Y., Wongpoowarak, P., Moolasarn, S., & Anderson., C. (2019). A transition from the BPharm to the PharmD degree in five selected countries. Pharmacy Practice, 17(3), 1611. https://doi.org/10.18549/PharmPract.2019.3.1611 

Tucker, S. A., Barela, E., Miller, R. L., & Podems, D. R. (2020). The Story of the AEA Competencies Task Force (2015–2018). New Directions for Evaluation, 2020(168), 31–48. https://doi.org/10.1002/ev.20439 

what industries employ evaluators—Google Search. (2024). 

What Is Leadership? | Definition by TechTarget. (n.d.). CIO. Retrieved March 17, 2024, from https://www.techtarget.com/searchcio/definition/leadership 

Wait, I Can’t Use p < 0.05?

June 1, 2024 by Jonah Hall

By Jake Working

How did we get here? Why are interpretations of significance testing and p-values under fire? What does this mean for you, the applied researcher who uses these methods?

The literature surrounding this topic is huge, so I will start to provide some background to these questions in this blog post by including a brief introduction to a few important articles. My name is Jake Working, and I am currently studying for my Ph.D. in Evaluation, Statistics, and Methodology at the University of Tennessee, Knoxville. Let’s learn together.

How Did We Get Here?

Understanding the history of null hypothesis significance testing and p-values is just as important as crafting the future of these analytical methods. In this section, I direct you to check out Lee Kennedy-Shaffer’s article “Before p < 0.05 to Beyond p < 0.05: Using History to Contextualize p-Values and Significance Testing” (2019).

Kennedy-Shaffer reminds us of the history of significance testing and the p-value, noting Sir Ronald Fisher’s popularization of p < 0.05 through historical and contextual lens. Fisher was advancing statistical methodology at the same time as statistic legends such as Karl Pearson (yes, that Pearson) and William Gosset (of Guinness “student” fame), who were all developing uses for significance testing. Fisher formed his suggested p < 0.05 as a simple cut-off of significance in 1925. His reasoning was simple: “p = 0.05, or 1 in 20, is 1.96 or nearly 2…deviations exceeding twice the standard deviation are thus formally regarded as significant” (Fisher, 1925, p. 47 in Kennedy-Shaffer, 2019, p. 84).

Criticisms and alternatives to interpretations to significance testing have existed since the onset of null hypothesis significance testing. These include Neyman-Pearson’s alpha (1933), Bayes’ inverse probability, and Fisher himself even challenged the field against a fixed level of significance (Kennedy-Shaffer, 2019, pp. 85-86). So, what’s the beef with p-values now? 

Laying Down the Law 

As the discussion on p-values and other flaws in statistical reporting seemed to rekindle in the mid-2010s, the American Statistical Association decided to provide the scientific and research community with grounded direction on p-values. In this section, I urge you to read the very short, but impactful “ASA Statement on p-Values: Context, Process, and Purpose” by Ronald Wasserstein and Nicole Lazar (2016).  

They articulated six simple principles on p-values: 

1. P-values can indicate how incompatible the data are with a specified statistical model 

2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone 

3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a scientific threshold 

4. Proper inference requires full reporting and transparency 

5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result 

6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis 

These principles urge the researcher to contextualize and completely understand their data and analysis methods, making useless bright lines such as p < 0.05. Rosnow and Rosenthal (1989) said it neatly: “…surely, God loves the .06 nearly as much as the .05” (p. 1277). 

Okay, so what do I do now? 

If you are a researcher, student, or just interested in statistical analysis, one thing you can do is to update your analytical habits. Check out this article by Wasserstein, Lazar, and Schirm: “Moving to a World Beyond ‘p < 0.05’” (2019) for context and suggestions. 

Included in their article is a mental framework to guide future use of these statistical methods they summarize into two sentences: “Accept uncertainty. Be thoughtful, open, and modest” (Wasserstein et al., 2019, p. 2). Their framework is helpful to set your mental state before delving into the eight pages of action items summarized from 43 different articles on this topic.  

Wasserstein et al. make it an easy read by summarizing each article into actionable bullet points and organizing the suggestions into five topic areas: 

1. Getting to a Post “p < 0.05” Era 

2. Interpreting and Using p 

3. Supplementing or Replacing p 

4. Adopting More Holistic Approaches 

5. Reforming Institutions: Changing Publication Policies and Statistical Education 

Call to Action 

As it would be impossible to summarize everything from these articles into one blog post, I urge you to read the three articles in this post. You will better understand p-values and become a better researcher, evaluator, and statistician because of it.  

1. “Before p < 0.05 to Beyond p < 0.05: Using History to Contextualize p-Values and Significance Testing” (Kennedy-Shaffer, 2019) 

2. “The ASA Statement on p-Values: Context, Process, and Purpose” (Wasserstein & Lazar, 2016) 

3. “Moving to a World Beyond p < 0.05” (Wasserstein et al., 2019) 

No need to abandon hypothesis testing and p-values, but be prepared to better understand these tools for what they are: statistical tools. 

References 

Kennedy-Shaffer L. (2019). Before p < 0.05 to Beyond p < 0.05: Using History to Contextualize p-Values and Significance Testing. The American Statistician, 73(Suppl 1), 82–90. https://doi.org/10.1080/00031305.2018.1537891  

Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284. 

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108  

Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a World Beyond “p< 0.05”. The American Statistician, 73(sup1), 1-19. https://doi.org/10.1080/00031305.2019.1583913