Synthetic Replication Games
In the context of a large replication effort of experimental work using LLMs as synthetic participants, the Talking to Machine team is organizing an in-person one-day event dedicated to adapting the original material of experiments published in top-tier journals to a standardised format which allows replicating them with a synthetic sample. The objective of the project is to systematically evaluate whether, and under which conditions, LLMs can be leveraged to augment or replace human samples in experimental work.
We are inviting researchers from diverse backgrounds and at all career stages to participate. During the event, participants will form small teams of 3-5 members, based on their research interests. Each team will be assigned a study from a leading journal. As part of the event, teams that complete the generation of the material for the synthetic replication will be invited to run a pilot replication of the study and present the results during a dedicated round table on LLMs in experimental social sciences included in the IMEBESS' main program.
Participants will be recognized as co-authors on the paper that will present aggregated results of multiple synthetic replications using a variety of LLMs.
What to Expect
Researchers participating in the event will join a small team of 3-5 members with shared research interests. Each team will engage in the synthetic replication of an experiment published in a top-tier journal and previously replicated with a human sample. Each synthetic replications will involve:
- The computational replication of the original results using the provided analysis script.
- Converting the original data collection instrument into a set of experimental prompts in a standardised format compatible with the Talking To Machine's platform to conduct synthetic experiments.
- Introduce into the data collection instrument the prompt adaptations required to conduct the experiment with a synthetic sample.
- Run a pilot of the synthetic replication using various LLMs and synthetic samples.
When
This event is part of IMEBESS 2025 and will run from 10:00 AM to 4:00 PM local time on 23 May 2025.
» Registration deadline: April 15th, 2025.
Registration
To register for the event at IMEBESS, please fill out the form below.