Organizing your Evaluation Data: The Importance of Having a Comprehensive Data Codebook

August 16, 2024 by Jonah Hall

By J.A. Morrow

Why a Data Codebook? 

One of my pet peeves is a disorganized project and inconsistency in how data are organized. For every project, whether it is an evaluation research or assessment project, I start developing a data codebook before I even begin data collection. When I take on a new project from an evaluation or assessment client, I first ask for their codebook or if they don’t have one then I create it for them. Why is this so important, you ask? Think of your codebook as your organizational tool and project history all rolled into one document. It contains everything about your project and greatly aids in getting everyone on your team organized and on the same page. Your clients (and your future self) will greatly appreciate this too!  

Your data codebook is a living document, it changes throughout the life of a project as you add new data, modify data, and make decisions throughout the course of the project. Not having a data codebook can lead to confusion and increase the chances of someone on your team making a mistake when analyzing data and disseminating information to your clients. Sadly, I have sat through presentations where a client points out a mistake or has a question about the data that can’t be answered by the evaluation team because they don’t have a record of what was done. Clients are never happy when this happens! 

What is in a Data Codebook? 

I usually include the following 9 things in my data codebooks: 

1. Name of the Evaluation Project 

2. Variable Names 

3. Variable Labels 

4. Value Labels 

5. Newly Created/Modified Variables (and how you created/modified these) 

6. Citations for Scales and Sources of Data for the Project 

7. Reliability of any Composite Items 

8. List of Datasets and Sample Size for Each 

9. Project Diary/Notebook 

I typically put the first 7 in one table, which I create in Microsoft Word. You can also create your codebook using Excel or any other analysis software package (e.g, SPSS, R). This first table provides details about all of the data for a project. As I make any changes to the datasets, I add any new variables that I create to this table and write up my decision making for any changes in the project diary/notebook section of my codebook. 

For the list of datasets and sample sizes I usually have that as a separate table at the end of my codebook. As I create a new dataset or project file I enter that information in this section of the codebook. I also include a brief description of what is contained in the new data file. I always organize this table by the most recent files first. 

Lastly, I include an extensive project diary/notebook at part of my codebook. For some projects these can be very long and have many team members adding to it so I typically will have this as a document link in the codebook. The document link takes team members to an external Google document where we all can write and edit information about what we are working on for the project and what decisions were made. I cannot overstate how important it is to have a detailed project diary/notebook for an evaluation project. It is especially useful as you are writing your reports for your client about what you did and why you did something in a particular way. Anytime I have a project meeting with my team or a meeting with my client I take notes in our project notebook. 

Additional Advice 

So, I hope I have provided some useful tips as you start the process of organizing your evaluation data. One last piece of advice….share this codebook with your client! At the end of a project, I give the codebook (minus the project notebook as that is internal to my team) and final datasets (sanitized at some level depending on the contract) to my client so they can continue to utilize the data for their program/organization. Empower your evaluation clients to better understand their data and how their data was processed! 

Resources 

12 Steps of Data Cleaning Handout:
https://www.dropbox.com/scl/fi/x2bf2t0q134p0cx4kvej0/TWELVE-STEPS-OF-DATA-CLEANING-BRIEF-HANDOUT-MORROW-2017.pdf?rlkey=lfrllz3zya83qzeny6ubwzvjj&dl=0 

https://datamgmtinedresearch.com/document

https://dss.princeton.edu/online_help/analysis/codebook.htm

https://ies.ed.gov/ncee/rel/regions/central/pdf/CE5.3.2-Guidelines-for-a-Codebook.pdf

https://libguides.library.kent.edu/SPSS/Codebooks

https://web.pdx.edu/~cgrd/codebk.htm

https://www.datafiles.samhsa.gov/get-help/codebooks/what-codebook

https://www.icpsr.umich.edu/web/ICPSR/cms/1983

https://www.medicine.mcgill.ca/epidemiology/joseph/pbelisle/CodebookCookbook/CodebookCookbook.pdf

https://www.slideshare.net/sl

The Conventional and Unconventional Places I’ve Found Research Ideas

August 1, 2024 by Jonah Hall

By Dr. Austin Boyd

Research is the backbone of the academic world. It provides us with a ​​​​better understanding of the world around us and allows knowledge to be passed on and built upon by future generations of researchers. Research may be conducted as​ a​ class project, larger end of course capstone or dissertation requirement, or as ​a ​regular part of many careers in and outside of academia. But where does one come up with a research idea? For some, finding inspiration for research projects is easy, resulting in a laundry list of ideas to pick and choose from. For others, coming up with even a single research idea might take longer than the research itself. Compound this with the fact that some fields have existed for decades, or even centuries, and it might feel as though there is nothing left to research. However, as daunting as it may seem, there are always more questions to be asked, you just need to ​​keep an open mind.

Project 1: A Student with a Question

My first research idea came about as conventionally as ​they​ come. I was a student with a question, and with the guidance of a professor, we came up with a research idea and then pursued it. Once upon a time, I took a statistics course on Item Response Theory (IRT). While sitting in class one day, we were discussing the underlying assumptions of IRT presented in Embretson and Reise’s Psychometric Methods: Item response theory for psychologists (2000). After class, I ​approached​ my professor with a question: “How does skewness impact measurement invariance?” Little did I know, this was a question she had always wondered herself, but never had the time to pursue. Over the next few weeks in office hours, we discussed ideas on how to address this question, and before long, she told me that she could provide me with data if I would be interested in exploring the topic further. Over the course of the next three years, she and I worked to test the robustness of this assumption, ​and​ ended up presenting our findings at two conferences and published them in the Journal of Applied Measurement (Boyd et al., ​​2020).  

Project 2: Friends Talking About Movies

Project 3: Watching YouTube to Avoid Schoolwork

My third research idea was born out of avoiding schoolwork. In one of my graduate courses, we had to develop and design an original survey on a topic of our choosing. There were many steps to this project, ​the first of which was ​​imply to propose an idea for the survey. Instead of doing that, I was watching people play video games on YouTube. After a while, I started wondering what makes online celebrities and influencers so popular, and after a quick Google search, I learned about a concept called parasocial relationships. These are the one-sided relationship​s​ that a viewer makes with a performer (Horton & Wohl, 1956). I kept digging into the topic and learned that people have been researching parasocial relationships and interactions for over 60 years, long before YouTube, or even the internet, existed. ​S​everal surveys had already been developed for ​understanding​​     ​ parasocial relationships and interactions with television personalities, TV characters, fictional characters, and even political candidates, but none for social media based celebrities and influencers. I decided that this would be the topic of my survey. Over the course of the semester, I developed my survey and put together a proposal for how I would pilot and assess the reliability and validity of the survey. I could have walked away from it once the course was over, but I came back a year later once I realized that this survey could actually be the basis for a real research project. As a result, my one class project became the basis for my entire dissertation and yielded two publications on the development and validity of the Parasocial Relationship in Social Media (PRISM) Survey (Boyd et al., 2022; Boyd et al. 2023).​ 

Research ideas are everywhere, even when it seems like there is nothing left to explore. And when it feels this way there are ​​five places I suggest taking a look:  

1. Prior literature – Prior literature if full of research ideas. Many publications include a section on future research ideas in the discussion, some of which are never fully explored. This can be a great place to start with a new research interest. 

2. Old class projects – Returning to an idea after being away from it can provide a new outlook that sparks a research idea. Coming back to an old project with the new knowledge gained from working on others can be revitalizing. 

3. Other researchers – Whether they be professors or peers, other researchers can be a great sounding board for ideas. Their knowledge and experiences can provide different points of view that can help inspire new project ideas. Some might even share ideas that they don’t have the time or interest in pursuing. 

4. Personal hobbies and interests – It might seem weird, but even personal interests can lead to research ideas. Without my interest in Marvel movies and YouTube, neither of my projects would have existed. 

5. ​​​Friends and family – Even if they don’t understand your research, sometimes talking to friends and family about it can spark new ideas. Their lack of knowledge on the subject can bring up questions that you never even thought about.​     ​ 

References:

​​​Boyd, A. T., Rocconi, L. M., & Morrow, J. A. (2024). Construct validation and measurement invariance of the Parasocial Relationships in Social Media Survey. PLoS ONE. 

Boyd, A. T., Morrow, J. A., & Rocconi, L. M. (2022). Development and validation of the Parasocial Relationship in Social Media Survey. The Journal of Social Media in Society, 11(2), 192-208. Available online: https://www.thejsms.org/index.php/JSMS/article/view/1085 

Boyd, A. T., & Rocconi, L. M. (2021). Formatting data for one and two mode undirected social network analysis. Practical Assessment, Research & Evaluation, 26(24). Available online: https://scholarworks.umass.edu/pare/vol26/iss1/24/ 

Boyd, A. T., Schmidt, K. M., & Bergeman, C. S. (2020). You know what they say about when you assume: Testing the robustness of invariant comparisons. Journal of Applied Measurement, 21(2), 190-209. 

Embretson, S. E., & Reise, S. (2000). Psychometric methods: Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum. 

Horton, D., & Wohl, R. R. (1956). Mass communication and parasocial interaction. Psychiatry, 19(3), 215–229. doi: 10.1080/00332747.1956.11023049