The use of online surveys in contemporary social science research has grown rapidly due to their many benefits such as cost-effectiveness and ability to yield insights into attitudes, experiences, and perceptions. Unlike more established methods such as pen-and-paper surveys, they enable complex setups like experimental designs and seamless integration of digital media content. But despite their user-friendliness, even seasoned researchers still face numerous challenges in creating online surveys. To showcase the versatility and common pitfalls of online surveying, Martin Emmer, Christian Strippel, and Roland Toth of the Methods Lab arranged the workshopIntroduction to Online Surveyson February 22, 2024.
Martin gave a presentation on the design and logic of online surveys.
In the first segment, Martin Emmer provided a theoretical overview of the design and logic of online surveys. He started by outlining the common challenges and benefits associated with interviewing, with a particular emphasis on social-psychological dynamics. Compared to online surveys, face-to-face interviews offer a more personal, engaging, and interactive experience, enabling interviewers to adjust questions and seek clarification of answers in real time. However, they can be time-consuming and expensive and may introduce biases such as the interviewer effect. On the other hand, the process of conducting online surveys presents its own set of challenges, such as limited control over the interview environment, a low drop-out threshold, and particularities connected with self-administration such as the need for detailed text-based instructions for respondents. Nevertheless, self-administered and computer-administered surveys boast numerous advantages, including cost-effectiveness, rapid data collection, the easy application of visuals and other stimuli, and accessibility to large and geographically dispersed populations. When designing an online survey, Martin stressed the importance of clear question wording, ethical considerations, and robust procedures to ensure voluntary participation and data protection.
Christian shared his insights on survey creation using online access panel providers.
In the second part of the workshop, Christian Strippel delved into the realm of online access panel providers, including the perks and pitfalls associated with utilizing them in survey creation. Panel providers serve as curated pools of potential survey participants managed by institutions, such as Bilendi/Respondi, YouGov, Cint, Civey, and the GESIS Panel. Panel providers oversee the recruitment and management processes, ensuring participants are matched with surveys relevant to their demographics and interests, while also handling survey distribution and data collection. While the use of online panels offers advantages such as accessing a broad participant pool, cost-efficiency, and streamlined sampling of specific sub-groups, they also have their limitations. Online panels are, for example, not entirely representative of the general population as they exclude non-internet users. Moreover, challenges arise from professional respondents such as so-called speeders who rush through surveys, and straight-liners who consistently choose the same response in matrix questions. Strategies to combat these issues include attention checks throughout the questionnaire, systematic exclusion of speeders and straight-liners, and quota-based screening. To conclude, Christian outlined what constitutes a good online panel provider, and shared valuable insights into how to plan a survey using one effectively.
Participants learned how to create their own survey using LimeSurvey during Roland’s live demo.
The third and final segment of the workshop featured a live demonstration by Roland Toth on how to set up an online survey using the open-source software LimeSurvey, which is hosted on the institute’s own servers. During this live demonstration, he created the very evaluation questionnaire administered to the workshop participants at the end of the workshop. Roland began by providing an overview of the general setup and relevant settings for survey creation. Subsequently, he demonstrated various methods of crafting questions with different scales, display conditions, and the incorporation of visual elements such as images. Throughout the demo, Roland addressed issues raised earlier in the first part of the workshop concerning language and phrasing, emphasizing rules for question-wording and why it is important to ask for one piece of information only per question. The live demonstration was wrapped up with a segment on viewing and exporting collected data. After letting the participants complete the evaluation form, the workshop concluded with a Q&A session.
Level: Beginner/Intermediate Category: Data Analysis
After being well received last year, we’re happy to announce the return of our workshop Programming and Data Analysis with R for its second edition. This two-day intensive workshop led by Roland Toth (WI) will take place on Wednesday, April 10, and Thursday, April 11, at the Weizenbaum Institute.
During the first day, attendees will receive comprehensive training in programming fundamentals, essential data wrangling techniques, and Markdown integration. The second day will center around data analysis, providing participants with the chance to engage directly with a dataset and address a research topic independently. A blend of concepts, coding techniques, and smaller practical tasks will be interspersed throughout both days to reinforce hands-on learning.
During my visit to the Center for Industry 4.0, I had the opportunity to participate in the pretest of the HoloLens study and learn more about augmented reality-based learning. The goal of the study, which is a collaboration between the research groups ofGergana Vladova(Education for the Digital World) and Martin Krzywdzinski (Working with Artificial Intelligence), is twofold. In the first part, the research groups investigate the effectiveness of different Augmented Reality (AR) designs on learning and compare them to traditional paper-based methods by using eye-tracking. In the second part, they focus on participants’ decision making and disruption management, guided by suggestions from an AI-assisted system. These participants can operate in either a team-based or hierarchical setting.
The cube travels along the conveyor belt, its screen showcasing lenses for participants to identify and sort out. The AR glasses used in the HoloLens study.
In the first part of the experiment, participants work in a simulated factory environment where they are tasked with producing lenses. The team uses either AR instructions or traditional paper instructions, depending on the experimental condition. The AR head-mounted display guides the team through tasks such as adjusting machine settings, sorting defective lenses, and other simulated problems. The same principle is used in the other experiments, except in this case, participants rely on paper-based learning instead of AR glasses.
In the second part of the experiment, participants apply what they learned in the first part, but without using the AR glasses or the paper instructions. In addition, the errors they must solve are different from those in the previous part. When presented with a problem, participants are expected to solve it collaboratively through effective communication and with the help of AI.
To measure performance, the study uses traditional metrics such as time and error rates. Between each round, knowledge tests in the form of a questionnaire are administered to assess participants’ recall and comprehension. The hypothesis is that process-integrated learning via Augmented Reality can enhance the learning process.
Nicolas Leins and Jana Gonnerman at the Centre for Industry 4.0 Potsdam.
The Weizenbaum Institute conducts research in a variety of ways. To provide an insight into the different research practices, from now on the Methods Lab will be presenting selected projects in longer features. For the first text in this series, Anna Hohwü-Christensen visited the Berlin Open Lab (BOL) to meet Ines Weigand and Corinna Canali from the research group Design, Diversity and New Commons.
I first met Ines and Corinna at the BOL in June, where they led the workshop Flushed Away: A Workshop on Disgust, Gender, and the Technical Object as part of the DGTF’s (Deutsche Gesellschaft für Design-Theorie und -Forschung) Design and Digital Justice Conference. Ines and Corinna are research associates affiliated with the Weizenbaum Institute through the research group Design, Diversity and New Commons. A group that, in turn, forms a cornerstone in the Design Research Lab initiative—a network of researchers and organizations that aim to bridge the gap between technological innovations and people’s real needs.
BOL is a dynamic and experimental space that brings together experts from design, engineering, the humanities, and maker communities. Based at the University of Arts Berlin (UdK) in Berlin-Charlottenburg, it acts as a convergence point for four institutions: Weizenbaum Institute, UdK, Technical University, and the Einstein Center Digital Future. Home to numerous transdisciplinary research projects, events, and conferences, BOL operates with the mission of application-oriented and inclusive design of human-technology interaction, and transparent, participatory research.
Following my initial visit, I decided to find out more about the lab and the part played by the Weizenbaum Institute within its intricate framework. During a comprehensive tour, I got to chat with Ines and Corinna not only about the space and its many diverse projects, but also about research and interdisciplinarity in design practices, and the importance of critical thinking in material making.
A map of the various spaces at BOL.
Anna: Can you tell me a bit about the Berlin Open Lab? What is it and how does it function?
Ines: BOL is an experimental space for transdisciplinary research projects at the intersection of technology, society, and arts. It has its own laboratory for digital-based production, smart material interfaces, and wearable computing plus a space for design research with augmented and virtual realities. There is this idea of shared resources and experiences, and flexible and agile working which they try to support with the spatial design of the space. Everything is movable and adaptable here, every group has their space to work in, but it is kind of fluent and changing. So when a new project comes in, you see how you can support them in their working structure, combining it with the spatial aspect. Behind this glass wall is the machine tool area.
And this is the working and event space. I can show you what kind of projects are in here as far as I know, but sometimes it also happens to me that, for example, when the BOL symposium was here, and there were so many projects presented… It’s like oh, I never knew that you are here, but officially, they also have this space.
[laughter]
Anna: I sometimes feel like that at the Weizenbaum Institute. There is just so much going on. Is the lab generally open to students who want to use this space?
Ines: There was, for example, this project by a student who was working on her master’s thesis. The project was about breast cancer patients who are losing one or two breasts. Apparently, there are only three shapes or so that you can go for if you are getting an artificial breast, so she tried to develop a process where you can scan the breast and get a prosthesis that is closer to the original shape. She applied for it, and was then able to use the space.
Anna: How many people work here on average on a full day?
Ines: It’s very different. There are maybe fifteen people here on a full day because not everyone is in here all the time. If people need to write, they are not here because it can get too noisy. It is more if they want to use the machines, tools, meet, and work on something together. And if you have silent work, like writing or something…
Corinna: That is why I am not sitting here.
[laughter]
Corinna: Never here…
Workspaces from the Design, Diversity and New Commons group.Various student projects on display.
Ines: This is the space from our research group Design Diversity, and New Commons. I am mainly sitting here with Michelle Christensen and Florian Conradi, and our two working students. We use experimental research methods that come out of design, so we are using mainly critical making as a method. This is an approach where you try to combine critical thinking or critical theory—which comes from the humanities—with material production. So the courses we give are always called something like Politics of Machines or Design and Conflict, but they all follow the same structure where students get to know a field of, let’s say, critical theory. It is mainly about technology, but our last course was on the relationship between the environment and humans. The students get a specific perspective with which they look at design and try to use design as a form of critical medium, a tool to bridge this way of critical thinking with material making. The objects that you see hereare from different courses that the students had. They build objects or artifacts that are not meant to work in a way so that you can scale them up and bring them to market, but that is more about exploring a way of thinking in materiality. So it is kind of like a critical thought translated into materiality. They are more like curious objects.
Ines and a student project on surveillance capitalism.Print-outs of data collected through cookies of user activity on different websites.
Ines: This was made by a student who looked into surveillance capitalism. He looked into cookies and how much data is constantly saved from you when you are surfing on the internet. As a project, he made this little printer box, where the data of the cookies that the website saved from you is printed in real-time. This, for example, is just 30 seconds of Yahoo!
Anna: Oh, wow.
Ines: And here you can see the different websites and how much data they are saving from you.
Anna: It is interesting to have it printed out like that.
Ines: Yeah. It is interesting because then you can actually see what cookies are. Because if you have a look at it, at some point, you are on a completely different website. And then Spotify turns up and it is like what, this is not the website I am on, but they are checking out everything you do on your computer, what programs you have open. And then you get it. What cookies actually are and what information they are getting from you.
Ines: And this is a project from Pablo. He wanted to find out about how we can get another approach or feeling about what is going on around us in the environment. He installed a CO2 sensor in a box and programmed it in a way so that it gives you a noise signal about how much CO2 is in the air. So you have another approach to what pollution is or how much pollution, or in this case, CO2, is in the air around us.
Anna: So it creates a sound depending on how much CO2 there is?
Ines: Exactly. The more CO2 there is, the more sound there is.
Anna: That is fascinating. And very creative!
Ines: Yeah. And this is a project where a student was looking into the weird fact that in some parts of the world, it is easier to get Coca-Cola than clean water. The sad story behind it is that clean water is somehow used to produce this Coca-Cola. So she made this as a critical object, a filter that turns Coca-Cola into water, filtering out mainly the sugar and other things, to highlight this weird fact. And that is the sense behind it. To highlight an issue, set up a specific critique, or come up with an alternative way of thinking. While the students are doing prototyping, they learn about technology and the power relations embedded in them, how they function, and how to use them and work with them by building up their own critiques and combining it with practical making. They have to deal with sensors and technology. That is the way they learn to approach this field—by really doing something combined with critical thinking.
Corinna: My part of the research group—that is now just Bianca Herlo and I—we don’t work here and we are not working specifically on design as a tool to make something. We are using design, artistic research, and visual culture research to analyze bias within the digital realm and technology to unpack issues that are mostly unseen. Because like with cookies, they are running in the background, and you need to have tools to visualize them to figure out how they function, how many they are, and how invasive they can be. And this is pretty much the Design Research Lab and what everyone is doing. So this [ground floor] is one part, but then there is the expanded part that mostly sits on the second floor. There are people working in artificial intelligence, in theory per se, in policy-making. It is a broad organism, the Design Research Lab.
Ines: And the BOL is more the practical side of it, a space that was introduced to allow people from a lot of different institutions to use shared tools, machines, knowledge, and open-source libraries. It is not an organization, it is a platform. For us, the main thing is to do research about design methods, and what they can contribute to lacks or errors in current research. Design research methods are often about trying to bridge between different concepts. That is the idea behind it.
A mannequin head wrapped in plastic.A student project on space farming.
Corinna: The research group is made up of the three of us [refers to student worker Selenay], Athena, and the heads. What we are doing sounds very different from when you actually see the projects that we are working on. Ines and I are both doing our PhDs. I am mostly working on gender bias within internet governance, and Ines is working on practical…
Ines: Practical bridging of the gap between humans and the environment.
Corinna: From the perspective of…
Ines: …the post-humanities.
Corinna: In a way, we found out that we are working on something that has a connection when we did a workshop.
Anna: I was there, yeah, I remember it.
Corinna: The workshop made us realize that there is a common ground. That is, I am working mostly on content moderation, so what gets excluded from the internet and what is allowed. Ines is working on bodily waste and the creation of waste, what the meaning and political significance is of making something into waste and having to throw it out. We are approaching subjects from very different perspectives. I come from visual culture, design, and art, and I am employing analytical methods to analyze image production and consumption online. It is two different ways of experimenting with different kinds of technologies. One is to do more with chemical and biochemical technology. Mine is more digital.
Corinna: Yes, and also moving across disciplines, which is why design research and artistic research have started to grow in the past few years… Because actually they can move across disciplines.
Ines: …and other disciplines struggle with that.
Corinna: Yes, other disciplines are very much constrained within their own boundaries. It is difficult to find people working on artificial intelligence that moves outside of data or computer science. When you are working within design research, it is kind of natural and organic that you grasp from all the disciplines that belong to that, also in some peripheral ways, not just directly. This is kind of what everyone is doing in this space, in a way. It gives you the tools to move across whatever.
Ines: We are using a process called research through design, so we are actively using the design process itself as an epistemological source. We are designing, and while we are designing we are reflecting on the design process and getting specific knowledge out of it. It is a very practical way of doing research.
Anna: Do you know if there are other labs that have this approach?
Ines: I think there is a whole movement that is trying to implement open labs. I don’t know if they are also doing research. I think that is the special thing here. It is an open, shared lab, but it is also an open shared lab where you are doing research on what is happening. Coming out of the maker movement, there are a lot of areas where people are trying to develop open labs, where you can share machines and access technology like laser cutting and 3D printing.
Corinna: There are similar things, but they are mostly focused and financed by industries. So the end goal is not to produce research, but to produce a commodity or something that can be turned into products. The main focus here is research and not producing something that becomes a mass product ready for market. It is to apply a critical, analytical approach to what you are putting out in the world.
Ines: Yeah, there are a lot of labs but not combined with a research focus. We are doing practical making, but also research on practical making through practical making. It is about what value the practical work has in research. There’s a lot of theory also, and for a long time, the material part was missing. Then the material turn came with the idea that we cannot be completely separated from the material world around us… And now they’re trying to find concepts of how to combine those again.
Corinna: At the beginning, it was mostly circumscribed to industrial and product design, and then it kind of started filtering through and moving within the design spectrum as a whole.
Ines: Yeah. And it is still a very young field. I think the whole design research field is still finding itself. It is not like there is one way and everyone is on the same page. There are a lot of different things going on now, people are trying things and having open discussions. It is still an experimental ground.
Research associates Corinna Canali and Ines Weigand.
The amalgamation of previously two separate groups, Design, Diversity, and New Commons is part of the Design Research Lab initiative based at the BOL at UdK Berlin. Led by principal investigator Gesche Joost, the group is comprised of research heads Michelle Christensen, Florian Conradi, and Bianca Herlo, research associates Ines Weigand and Corinna Canali, and student assistants Athena Grandis and Selenay Kiray.
Surveys are an important method for data collection. Whether it is for conducting internal assessments, gathering feedback, or collecting valuable research data, a reliable survey tool is an integral piece in the methodological toolkit of any researcher. Using different survey tools for different projects leads to differences in the quality of data collection and unnecessary licensing costs. In order to find a more sustainable solution, the Methods Lab assessed some of the most popular survey tools with the aim of finding the ideal one to cater to the specific needs of the Weizenbaum Institute’s research groups and administrative departments. Important to us was to select a user-friendly, open-source survey tool suitable for research that can be hosted on our own servers.
In this blog post, we introduce our choice: LimeSurvey. It is a free, open-source survey software with a strong commitment to data protection. It offers a versatile platform for data collection, making it ideal for researchers, academic institutions, and organizations of all sizes. In doing so, we hope that the insights from our survey tool comparison will prove useful to researchers and institutions beyond our own.
Here are some of the distinctive advantages that we were able to identify, making LimeSurvey a compelling choice for research and data collection:
Cost-Effective and Open Source: LimeSurvey is open source, meaning, it is available for free when hosted on your own servers, thereby eliminating the need for costly licensing fees.
Data Protection: LimeSurvey prioritizes data privacy – a particular advantage appreciated by our IT department due to its compliance with the GDPR. Its servers are strategically located in Germany and Finland, ensuring adherence to stringent European data protection regulations.
User-Friendly Integration: LimeSurvey seamlessly integrates with existing user accounts, simplifying the onboarding process without requiring additional account setup.
Suitable for Research: LimeSurvey is designed with research needs in mind. It offers a wide range of features, including unlimited projects and administrators/accounts. This flexibility makes it suitable for both simple and complex research projects.
No Artificial Limits: LimeSurvey imposes no artificial limitations on user accounts, participants, or projects.
Data is an invaluable asset for scientific research. However, accessing platform data for academic purposes has become increasingly challenging, particularly with the closure of free access to APIs like Twitter’s. Recognizing the significance of data accessibility for research, the Weizenbaum Institute organized the workshop Datenzugang für die Forschung – Der Digital Services Act (DSA)in collaboration with the European New School of Digital Studies (ENS) to explore the potential of the upcoming Digital Services Act (DSA) in facilitating data access for academic research.
The DSA is set to bring about improvements in data access for researchers under Article 40. However, the DSA’s regulations must be thoughtfully implemented at the national level to achieve these goals fully. With the closure of free access to Twitter’s API, there is an urgency to find robust solutions to enable researchers to access platform data for scientific inquiry. The DSA, expected to come into force in February 2024, holds promises to provide avenues for researchers to obtain the data they need for their academic research. Still, it also brings about its own set of challenges.
The workshop aimed to foster an open forum where researchers from diverse disciplines, particularly those who work or plan to work with platform data, could come together to provide recommendations for the effective implementation of the DSA. Organized by Ulrike Klinger (ENS) and Jakob Ohme (WI) and supported by the Stiftung Mercator, the workshop addressed crucial questions surrounding data access requests, eligible data, and the verification process by authorities and platforms.
The workshop started with a welcoming address from Ulrike Klinger. Jakob Ohme then provided an overview of the DSA’s Article 40, shedding light on its potential implications for researchers. This was followed by presentations on the DSA’s implementation in Germany by Gökhan Cetintas from the Bundesministerium für Digitales und Verkehr and Andrea Sanders-Winter from the Bundesnetzagentur, who offered insights into the data access rules under the DSA.
After a coffee break, Jessica Gabriele Walter from Aarhus University presented on DSA40 and scholarly networks in other EU countries, providing a broader perspective on data access challenges and solutions. Richard Kuchta from Democracy Reporting International later delved into “The Data Access Problem” and emphasized the necessity of a vetting process to ensure data security and accuracy.
The latter part of the workshop involved group work in which participants engaged in the discussion and expansion of a policy paper draft prepared by the Weizenbaum Institute and ENS, based on inputs from an early expert round. The goal was to develop actionable recommendations that would benefit the research community in Germany and the EU. Breakout sessions centered on topics like “Vetting Access,” “Access Modes,” and “Infrastructure,” allowing participants to delve deeper into specific aspects of data access.
The workshop brought together an interdisciplinary group of researchers with a shared vision: enabling access to platform data for academic purposes. By combining their expertise and perspectives, participants crafted recommendations for the effective implementation of the DSA, ensuring that data access for research remains equitable and secure. As the DSA comes into force and takes shape, the outcomes of this workshop are expected to serve as a significant step forward in fostering inclusive dialogue on the future of data accessibility.
We are excited to announce the launch of the Weizenbaum Panel Data Explorer, an interactive website developed by Methods Lab member Roland Toth. The Data Explorer allows you to browse and analyze survey results from the annual survey conducted by the Weizenbaum Panel on media use, political participation, civic norms, and more. In the spirit of open science, it not only presents research data, but also in an easy-to-use manner.
The Weizenbaum Panel aims to shed light on the complex relationship between the digital realm and political engagement. By examining phenomena such as hate speech and fake news, as well as the active commitment to a democratic culture of debate, the telephone survey offers invaluable insights into the ever-evolving dynamics of citizen participation in Germany.
With the launch of Data Explorer, you can explore this comprehensive dataset and gain a deeper understanding of Germany’s social and political landscape. The platform offers six categories: social media platform use, political attitudes, civic norms, political participation, and online civic intervention. Each category presents a unique perspective, allowing you to examine specific aspects of Germany’s social and political fabric.
The Weizenbaum Panel Data Explorer interface
To begin your exploration, simply select a category that piques your interest. Within each category, you will find a selection of questions to delve into. Whether you want to gauge the political news media consumption of the German public, analyze trends in the use of video platforms such as TikTok and Instagram, or find out how often people discuss political issues at work, or with friends and family, the Data Explorer will assist you in this endeavor.
For a nuanced understanding of how different groups within the population engage in social and political activities, you can group the data output by selecting the demographic factors gender, age, level of education, or residence. Moreover, to enhance your experience and facilitate data sharing, you can download any graph in .png format. Each graph includes the question, answering options, and grouping, providing a comprehensive visual representation of the desired data.
A graph downloaded in .png format.
The Weizenbaum Data Explorer was developed in Python/Jupyterhub and deployed using Voilà, which are all open-source. It is hosted on Weizenbaum Institute servers, which ensures adequate data protection. This is not the case for typical solutions such as using R Shiny and the deployment platform shinyapps.io. The Data Explorer will be expanded continuously – for example, the fourth wave of the Weizenbaum Panel will be integrated soon.
Whether you’re a researcher, journalist, student, or simply someone curious about Germany’s social and political landscape, the Weizenbaum Panel Data Explorer equips you with the tools to visualize data effortlessly. Happy exploring!
Manage Cookie Consent
To provide the best experiences, we use cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
