The use of online surveys in contemporary social science research has grown rapidly due to their many benefits such as cost-effectiveness and ability to yield insights into attitudes, experiences, and perceptions. Unlike more established methods such as pen-and-paper surveys, they enable complex setups like experimental designs and seamless integration of digital media content. But despite their user-friendliness, even seasoned researchers still face numerous challenges in creating online surveys. To showcase the versatility and common pitfalls of online surveying, Martin Emmer, Christian Strippel, and Roland Toth of the Methods Lab arranged the workshopIntroduction to Online Surveyson February 22, 2024.
Martin gave a presentation on the design and logic of online surveys.
In the first segment, Martin Emmer provided a theoretical overview of the design and logic of online surveys. He started by outlining the common challenges and benefits associated with interviewing, with a particular emphasis on social-psychological dynamics. Compared to online surveys, face-to-face interviews offer a more personal, engaging, and interactive experience, enabling interviewers to adjust questions and seek clarification of answers in real time. However, they can be time-consuming and expensive and may introduce biases such as the interviewer effect. On the other hand, the process of conducting online surveys presents its own set of challenges, such as limited control over the interview environment, a low drop-out threshold, and particularities connected with self-administration such as the need for detailed text-based instructions for respondents. Nevertheless, self-administered and computer-administered surveys boast numerous advantages, including cost-effectiveness, rapid data collection, the easy application of visuals and other stimuli, and accessibility to large and geographically dispersed populations. When designing an online survey, Martin stressed the importance of clear question wording, ethical considerations, and robust procedures to ensure voluntary participation and data protection.
Christian shared his insights on survey creation using online access panel providers.
In the second part of the workshop, Christian Strippel delved into the realm of online access panel providers, including the perks and pitfalls associated with utilizing them in survey creation. Panel providers serve as curated pools of potential survey participants managed by institutions, such as Bilendi/Respondi, YouGov, Cint, Civey, and the GESIS Panel. Panel providers oversee the recruitment and management processes, ensuring participants are matched with surveys relevant to their demographics and interests, while also handling survey distribution and data collection. While the use of online panels offers advantages such as accessing a broad participant pool, cost-efficiency, and streamlined sampling of specific sub-groups, they also have their limitations. Online panels are, for example, not entirely representative of the general population as they exclude non-internet users. Moreover, challenges arise from professional respondents such as so-called speeders who rush through surveys, and straight-liners who consistently choose the same response in matrix questions. Strategies to combat these issues include attention checks throughout the questionnaire, systematic exclusion of speeders and straight-liners, and quota-based screening. To conclude, Christian outlined what constitutes a good online panel provider, and shared valuable insights into how to plan a survey using one effectively.
Participants learned how to create their own survey using LimeSurvey during Roland’s live demo.
The third and final segment of the workshop featured a live demonstration by Roland Toth on how to set up an online survey using the open-source software LimeSurvey, which is hosted on the institute’s own servers. During this live demonstration, he created the very evaluation questionnaire administered to the workshop participants at the end of the workshop. Roland began by providing an overview of the general setup and relevant settings for survey creation. Subsequently, he demonstrated various methods of crafting questions with different scales, display conditions, and the incorporation of visual elements such as images. Throughout the demo, Roland addressed issues raised earlier in the first part of the workshop concerning language and phrasing, emphasizing rules for question-wording and why it is important to ask for one piece of information only per question. The live demonstration was wrapped up with a segment on viewing and exporting collected data. After letting the participants complete the evaluation form, the workshop concluded with a Q&A session.
Level: Beginner/Intermediate Category: Data Analysis
After being well received last year, we’re happy to announce the return of our workshop Programming and Data Analysis with R for its second edition. This two-day intensive workshop led by Roland Toth (WI) will take place on Wednesday, April 10, and Thursday, April 11, at the Weizenbaum Institute.
During the first day, attendees will receive comprehensive training in programming fundamentals, essential data wrangling techniques, and Markdown integration. The second day will center around data analysis, providing participants with the chance to engage directly with a dataset and address a research topic independently. A blend of concepts, coding techniques, and smaller practical tasks will be interspersed throughout both days to reinforce hands-on learning.
During my visit to the Center for Industry 4.0, I had the opportunity to participate in the pretest of the HoloLens study and learn more about augmented reality-based learning. The goal of the study, which is a collaboration between the research groups ofGergana Vladova(Education for the Digital World) and Martin Krzywdzinski (Working with Artificial Intelligence), is twofold. In the first part, the research groups investigate the effectiveness of different Augmented Reality (AR) designs on learning and compare them to traditional paper-based methods by using eye-tracking. In the second part, they focus on participants’ decision making and disruption management, guided by suggestions from an AI-assisted system. These participants can operate in either a team-based or hierarchical setting.
The cube travels along the conveyor belt, its screen showcasing lenses for participants to identify and sort out. The AR glasses used in the HoloLens study.
In the first part of the experiment, participants work in a simulated factory environment where they are tasked with producing lenses. The team uses either AR instructions or traditional paper instructions, depending on the experimental condition. The AR head-mounted display guides the team through tasks such as adjusting machine settings, sorting defective lenses, and other simulated problems. The same principle is used in the other experiments, except in this case, participants rely on paper-based learning instead of AR glasses.
In the second part of the experiment, participants apply what they learned in the first part, but without using the AR glasses or the paper instructions. In addition, the errors they must solve are different from those in the previous part. When presented with a problem, participants are expected to solve it collaboratively through effective communication and with the help of AI.
To measure performance, the study uses traditional metrics such as time and error rates. Between each round, knowledge tests in the form of a questionnaire are administered to assess participants’ recall and comprehension. The hypothesis is that process-integrated learning via Augmented Reality can enhance the learning process.
Nicolas Leins and Jana Gonnerman at the Centre for Industry 4.0 Potsdam.
We are excited to announce the Methods Lab’s first workshop of the year, “Introduction to Online Surveys“, which will take place on Thursday, February 22. This workshop will be conducted both at the Weizenbaum Institute and online, and is open to Weizenbaum Institute members as well as external participants. Led by members of the Methods Lab, Martin Emmer, Christian Strippel, and Roland Toth, the workshop will focus on the use of online surveys in the context of social science research, providing participants with a theoretical foundation as well as a hands-on guide. We will cover aspects such as the logic and design of online surveys, how to work with access panel providers, and demonstrate how to effectively set up an online survey using the versatile survey tool LimeSurvey. Crucial topics such as ethics and data protection will also be discussed.
For detailed information about the workshop, please visit our program page. We look forward to your participation!
Digital and computational data collection and analysis methods such as mobile/internet tracking, experience sampling, web scraping, text mining, machine learning, and image recognition have become more relevant than ever in the social sciences. While these methods enable new avenues of inquiry, they also present many challenges. It is important to share and discuss research, experiences, and challenges surrounding these methods with other researchers to exchange ideas and to learn from experiences.
For this reason, Roland Toth from the Methods Lab and research fellow Douglas Parry organized the Digital Methods Colloquium that took place on December 7 at the Weizenbaum Institute. They invited researchers from all over Germany who had used such methods before. The focus lied on sharing not only successes, but – even more so – the challenges that they had experienced in the research process.
Fenne Große Deters (U of Potsdam) talking about the effects of smartphone use on sleep quality
In the first part of the colloquium, participants presented recent or past research projects for which they had used digital methods. The presentations covered various methods, including experience sampling, mobile logging/tracking, multimodal content classification, network analysis, and large language models. All presentations were received very well and led to high engagement with many questions and exchanges from the participants.
The second part of the colloquium was designed to facilitate interactive discussion and knowledge sharing among the participants. They were assigned to one of two discussion groups that focused on either data collection or data analysis in the context of digital methods. In each group, participants followed prompts and discussed urgent issues and possible solutions, which they then visualized using posters. Finally, both groups sat together and presented the posters to each other, leading to a final discussion. After a short wrap-up, some participants joined the hosts at the Christmas Market for a well-deserved hot beverage.
Patrick Zerrer (U of Bremen) talking about mobile usage patterns of young political activists
The hosts would like to thank all participants for attending and engaging in the Digital Methods Colloquium. Bringing together researchers from different fields demonstrated that there are more commonalities than differences when it comes to the challenging and exciting field of digital methods. We are looking forward to more exchange and, possibly, Part 2 of the Digital Methods Colloquium sometime in the future.
On November 30th, 2023, the Methods Lab organized a workshop on quantitative text analysis. The workshop was conducted by Douglas Parry (Stellenbosch University) and covered the whole process of text analysis from data preparation to the visualization of sentiments or topics identified.
In the first half of the workshop, Douglas covered the first steps involved in text analysis, such as tokenization (the transformation of texts into smaller parts like single words or consecutive words), the removal of “stop words” (words that do not contain meaningful information), and the aggregation of content by meta-information (authors, books, chapters, etc.). Apart from the investigation of the frequency with which terms occur, sentiment analysis using existing dictionaries was also addressed. This technique involves assigning values to each word representing certain targeted characteristics (e.g., emotionality/polarity), which in turn allows for comparing overall sentiments between different corpora. Finally, the visualization of word occurrences and sentiments was covered. After this introduction, participants had the chance to apply their knowledge using the programming language R by solving tasks with texts Douglas provided.
Douglas Parry goes through steps necessary to prepare for text analysis.
In the second half of the workshop, Douglas focused on different methods of topic modeling, which ultimately attempt to assign texts to latent topics based on the words they contain. In comparison to simpler procedures covered in the first half of the workshop, topic models can also consider the context of words within the texts. Specifically, Douglas introduced participants to Latent Dirichlet Allocation (LDA), Correlated Topic Modeling (CTM), and Structural Topic Modeling (STM). One of the most important decisions to be made for any such model is the number of topics to emerge: too few may dilute nuances within topics and too many may lead to redundancies. The visualization and – most importantly – limitations of topic modeling were also discussed before participants performed topic modeling themselves with the data provided earlier. Finally, Douglas concluded with a summary of everything covered and an overview of advanced subjects in text analysis.
Participants solve practical exercises.
The workshop was very well-received and prepared all participants for text analysis in the future. Douglas balanced lecture-style sections and well-prepared, hands-on application very well and provided all materials in a way that participants could focus on the tasks at hand, while following a logical structure throughout. We would like to thank him for this great introduction to text analysis!
The Weizenbaum Institute conducts research in a variety of ways. To provide an insight into the different research practices, from now on the Methods Lab will be presenting selected projects in longer features. For the first text in this series, Anna Hohwü-Christensen visited the Berlin Open Lab (BOL) to meet Ines Weigand and Corinna Canali from the research group Design, Diversity and New Commons.
I first met Ines and Corinna at the BOL in June, where they led the workshop Flushed Away: A Workshop on Disgust, Gender, and the Technical Object as part of the DGTF’s (Deutsche Gesellschaft für Design-Theorie und -Forschung) Design and Digital Justice Conference. Ines and Corinna are research associates affiliated with the Weizenbaum Institute through the research group Design, Diversity and New Commons. A group that, in turn, forms a cornerstone in the Design Research Lab initiative—a network of researchers and organizations that aim to bridge the gap between technological innovations and people’s real needs.
BOL is a dynamic and experimental space that brings together experts from design, engineering, the humanities, and maker communities. Based at the University of Arts Berlin (UdK) in Berlin-Charlottenburg, it acts as a convergence point for four institutions: Weizenbaum Institute, UdK, Technical University, and the Einstein Center Digital Future. Home to numerous transdisciplinary research projects, events, and conferences, BOL operates with the mission of application-oriented and inclusive design of human-technology interaction, and transparent, participatory research.
Following my initial visit, I decided to find out more about the lab and the part played by the Weizenbaum Institute within its intricate framework. During a comprehensive tour, I got to chat with Ines and Corinna not only about the space and its many diverse projects, but also about research and interdisciplinarity in design practices, and the importance of critical thinking in material making.
A map of the various spaces at BOL.
Anna: Can you tell me a bit about the Berlin Open Lab? What is it and how does it function?
Ines: BOL is an experimental space for transdisciplinary research projects at the intersection of technology, society, and arts. It has its own laboratory for digital-based production, smart material interfaces, and wearable computing plus a space for design research with augmented and virtual realities. There is this idea of shared resources and experiences, and flexible and agile working which they try to support with the spatial design of the space. Everything is movable and adaptable here, every group has their space to work in, but it is kind of fluent and changing. So when a new project comes in, you see how you can support them in their working structure, combining it with the spatial aspect. Behind this glass wall is the machine tool area.
And this is the working and event space. I can show you what kind of projects are in here as far as I know, but sometimes it also happens to me that, for example, when the BOL symposium was here, and there were so many projects presented… It’s like oh, I never knew that you are here, but officially, they also have this space.
[laughter]
Anna: I sometimes feel like that at the Weizenbaum Institute. There is just so much going on. Is the lab generally open to students who want to use this space?
Ines: There was, for example, this project by a student who was working on her master’s thesis. The project was about breast cancer patients who are losing one or two breasts. Apparently, there are only three shapes or so that you can go for if you are getting an artificial breast, so she tried to develop a process where you can scan the breast and get a prosthesis that is closer to the original shape. She applied for it, and was then able to use the space.
Anna: How many people work here on average on a full day?
Ines: It’s very different. There are maybe fifteen people here on a full day because not everyone is in here all the time. If people need to write, they are not here because it can get too noisy. It is more if they want to use the machines, tools, meet, and work on something together. And if you have silent work, like writing or something…
Corinna: That is why I am not sitting here.
[laughter]
Corinna: Never here…
Workspaces from the Design, Diversity and New Commons group.Various student projects on display.
Ines: This is the space from our research group Design Diversity, and New Commons. I am mainly sitting here with Michelle Christensen and Florian Conradi, and our two working students. We use experimental research methods that come out of design, so we are using mainly critical making as a method. This is an approach where you try to combine critical thinking or critical theory—which comes from the humanities—with material production. So the courses we give are always called something like Politics of Machines or Design and Conflict, but they all follow the same structure where students get to know a field of, let’s say, critical theory. It is mainly about technology, but our last course was on the relationship between the environment and humans. The students get a specific perspective with which they look at design and try to use design as a form of critical medium, a tool to bridge this way of critical thinking with material making. The objects that you see hereare from different courses that the students had. They build objects or artifacts that are not meant to work in a way so that you can scale them up and bring them to market, but that is more about exploring a way of thinking in materiality. So it is kind of like a critical thought translated into materiality. They are more like curious objects.
Ines and a student project on surveillance capitalism.Print-outs of data collected through cookies of user activity on different websites.
Ines: This was made by a student who looked into surveillance capitalism. He looked into cookies and how much data is constantly saved from you when you are surfing on the internet. As a project, he made this little printer box, where the data of the cookies that the website saved from you is printed in real-time. This, for example, is just 30 seconds of Yahoo!
Anna: Oh, wow.
Ines: And here you can see the different websites and how much data they are saving from you.
Anna: It is interesting to have it printed out like that.
Ines: Yeah. It is interesting because then you can actually see what cookies are. Because if you have a look at it, at some point, you are on a completely different website. And then Spotify turns up and it is like what, this is not the website I am on, but they are checking out everything you do on your computer, what programs you have open. And then you get it. What cookies actually are and what information they are getting from you.
Ines: And this is a project from Pablo. He wanted to find out about how we can get another approach or feeling about what is going on around us in the environment. He installed a CO2 sensor in a box and programmed it in a way so that it gives you a noise signal about how much CO2 is in the air. So you have another approach to what pollution is or how much pollution, or in this case, CO2, is in the air around us.
Anna: So it creates a sound depending on how much CO2 there is?
Ines: Exactly. The more CO2 there is, the more sound there is.
Anna: That is fascinating. And very creative!
Ines: Yeah. And this is a project where a student was looking into the weird fact that in some parts of the world, it is easier to get Coca-Cola than clean water. The sad story behind it is that clean water is somehow used to produce this Coca-Cola. So she made this as a critical object, a filter that turns Coca-Cola into water, filtering out mainly the sugar and other things, to highlight this weird fact. And that is the sense behind it. To highlight an issue, set up a specific critique, or come up with an alternative way of thinking. While the students are doing prototyping, they learn about technology and the power relations embedded in them, how they function, and how to use them and work with them by building up their own critiques and combining it with practical making. They have to deal with sensors and technology. That is the way they learn to approach this field—by really doing something combined with critical thinking.
Corinna: My part of the research group—that is now just Bianca Herlo and I—we don’t work here and we are not working specifically on design as a tool to make something. We are using design, artistic research, and visual culture research to analyze bias within the digital realm and technology to unpack issues that are mostly unseen. Because like with cookies, they are running in the background, and you need to have tools to visualize them to figure out how they function, how many they are, and how invasive they can be. And this is pretty much the Design Research Lab and what everyone is doing. So this [ground floor] is one part, but then there is the expanded part that mostly sits on the second floor. There are people working in artificial intelligence, in theory per se, in policy-making. It is a broad organism, the Design Research Lab.
Ines: And the BOL is more the practical side of it, a space that was introduced to allow people from a lot of different institutions to use shared tools, machines, knowledge, and open-source libraries. It is not an organization, it is a platform. For us, the main thing is to do research about design methods, and what they can contribute to lacks or errors in current research. Design research methods are often about trying to bridge between different concepts. That is the idea behind it.
A mannequin head wrapped in plastic.A student project on space farming.
Corinna: The research group is made up of the three of us [refers to student worker Selenay], Athena, and the heads. What we are doing sounds very different from when you actually see the projects that we are working on. Ines and I are both doing our PhDs. I am mostly working on gender bias within internet governance, and Ines is working on practical…
Ines: Practical bridging of the gap between humans and the environment.
Corinna: From the perspective of…
Ines: …the post-humanities.
Corinna: In a way, we found out that we are working on something that has a connection when we did a workshop.
Anna: I was there, yeah, I remember it.
Corinna: The workshop made us realize that there is a common ground. That is, I am working mostly on content moderation, so what gets excluded from the internet and what is allowed. Ines is working on bodily waste and the creation of waste, what the meaning and political significance is of making something into waste and having to throw it out. We are approaching subjects from very different perspectives. I come from visual culture, design, and art, and I am employing analytical methods to analyze image production and consumption online. It is two different ways of experimenting with different kinds of technologies. One is to do more with chemical and biochemical technology. Mine is more digital.
Corinna: Yes, and also moving across disciplines, which is why design research and artistic research have started to grow in the past few years… Because actually they can move across disciplines.
Ines: …and other disciplines struggle with that.
Corinna: Yes, other disciplines are very much constrained within their own boundaries. It is difficult to find people working on artificial intelligence that moves outside of data or computer science. When you are working within design research, it is kind of natural and organic that you grasp from all the disciplines that belong to that, also in some peripheral ways, not just directly. This is kind of what everyone is doing in this space, in a way. It gives you the tools to move across whatever.
Ines: We are using a process called research through design, so we are actively using the design process itself as an epistemological source. We are designing, and while we are designing we are reflecting on the design process and getting specific knowledge out of it. It is a very practical way of doing research.
Anna: Do you know if there are other labs that have this approach?
Ines: I think there is a whole movement that is trying to implement open labs. I don’t know if they are also doing research. I think that is the special thing here. It is an open, shared lab, but it is also an open shared lab where you are doing research on what is happening. Coming out of the maker movement, there are a lot of areas where people are trying to develop open labs, where you can share machines and access technology like laser cutting and 3D printing.
Corinna: There are similar things, but they are mostly focused and financed by industries. So the end goal is not to produce research, but to produce a commodity or something that can be turned into products. The main focus here is research and not producing something that becomes a mass product ready for market. It is to apply a critical, analytical approach to what you are putting out in the world.
Ines: Yeah, there are a lot of labs but not combined with a research focus. We are doing practical making, but also research on practical making through practical making. It is about what value the practical work has in research. There’s a lot of theory also, and for a long time, the material part was missing. Then the material turn came with the idea that we cannot be completely separated from the material world around us… And now they’re trying to find concepts of how to combine those again.
Corinna: At the beginning, it was mostly circumscribed to industrial and product design, and then it kind of started filtering through and moving within the design spectrum as a whole.
Ines: Yeah. And it is still a very young field. I think the whole design research field is still finding itself. It is not like there is one way and everyone is on the same page. There are a lot of different things going on now, people are trying things and having open discussions. It is still an experimental ground.
Research associates Corinna Canali and Ines Weigand.
The amalgamation of previously two separate groups, Design, Diversity, and New Commons is part of the Design Research Lab initiative based at the BOL at UdK Berlin. Led by principal investigator Gesche Joost, the group is comprised of research heads Michelle Christensen, Florian Conradi, and Bianca Herlo, research associates Ines Weigand and Corinna Canali, and student assistants Athena Grandis and Selenay Kiray.
The Methods Lab is excited to welcome its first research fellow who arrived at the Weizenbaum Institute on November 20: Douglas Parry from Stellenbosch University, South Africa. His research focus lies on Socio-Informatics in the area of Communication Science, Human-Computer Interaction, and Media/CyberPsychology.
Roland Toth, Christian Strippel, and Douglas Parry (left to right).
During his 4-week stay, Douglas Parry will contribute to work at the Methods Lab in different ways. On November 30, he will hold the workshop A Practical Introduction to Text Analysis, where he covers all important steps, from pre-processing text to visualizing results of topic modeling in a single day. On December 7, he will host a Digital Methods Colloquium together with Roland Toth, where German researchers focusing on digital methods will get together, present recent work, and discuss challenges and opportunities in the field.
Douglas Parry discussing problems when measuring digital behavior in the Weizenbaum Fellow Colloquium.
Furthermore, Douglas Parry is collaborating on two research projects with the Methods Lab during his stay, both of which involve the processing of complex data surrounding smartphone usage that were collected using multiple methods earlier this year.
The Methods Lab is happy to host Douglas Parry and is looking forward to the results of this exciting partnership – stay tuned!
We are excited to announce that Methods Lab lead Christian Strippel organizes a panel on “Data, Archive & Tool Demos” at the Annual Conference of the German Communication Association (DGPuK) on March 13-15, 2024, in Erfurt. The corresponding Call for Contributions can be found here: https://www.dgpuk2024.de/sonderfenster/
Similar to the “Tool Demos” at international conferences, the panel serves as a forum for sharing reusable research data, databases, collections, archives, as well as tools and R packages with a wider academic audience. This initiative builds on the success of the “Research Software for Communication and Media Studies” panel in 2019, but this time aiming to enhance the development, provision, and utilization of research infrastructures and resources in German-speaking communication and media research in general.
Colleagues who wish to present data, archives, or tools at the panel are invited to submit a short abstract (200-300 words), with relevant links or screenshots, tochristian.strippel@weizenbaum-institut.de by the submission deadline of November 30, 2023. To be eligible for submission, your tool or resource should not have been previously featured in the research tools panel 2019 or the special issue in Publizistik. It should be openly available for scholarly reuse and not operated for commercial purposes.
For more information and submission guidelines, please visit this page.
We are eager to announce our upcoming workshop, “A Practical Introduction to Text Analysis“, on Thursday, November 30, at the Weizenbaum Institute. Led by visiting fellow Dr. Douglas Parry(Stellenbosch University, South Africa), this workshop offers a comprehensive introduction to text analysis using the R programming language. Topics covered include text pre-processing (formats, tokenization, stemming, stop words, regex), dictionary analysis (lexicons, tf-idf, sentiment), topic modeling (LDA, CTM, STM), and data visualization. By the end of the workshop, participants will be equipped to tackle real-world text-mining tasks and have a solid foundation to move on to more advanced analysis techniques. While a basic understanding of R programming is anticipated, prior experience in text analysis is not necessary.
For more details about the workshop, visit our program page. We look forward to your participation!
Manage Cookie Consent
To provide the best experiences, we use cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
