Title: | Behavioural Studies of Large Language Models |
---|---|
Description: | Efficient way to design and conduct psychological experiments for testing the performance of large language models. It simplifies the process of setting up experiments and data collection via language models’ API, facilitating a smooth workflow for researchers in the field of machine behaviour. |
Authors: | Xufeng Duan, Shixuan Li, Zhenguang Cai |
Maintainer: | Xufeng Duan <[email protected]> |
License: | LGPL-3 |
Version: | 1.2.8 |
Built: | 2025-02-07 06:13:23 UTC |
Source: | https://github.com/xufengduan/macbehaviour |
Defines the experiment setup based on the stimuli loaded.
experimentDesign(data, session = 1, randomItem = FALSE, randomEvent = FALSE)
experimentDesign(data, session = 1, randomItem = FALSE, randomEvent = FALSE)
data |
A data frame that has been processed through the 'loadData' function, containing the experimental items and their attributes. |
session |
An integer indicating how many sessions (the whole set of trials) should be run. Default is 1, meaning no repetition. |
randomItem |
A logical indicating whether the Item should be randomized. Default is FALSE, meaning trials will occur in the order provided. |
randomEvent |
A logical indicating whether the Event should be randomized. Default is FALSE, meaning trials will occur in the order provided. |
A data.frame with the designed structure for the experiment, including any repetitions and randomizations as specified. Each row corresponds to a single trial or instance in the experiment.And it will display the type of experiment with the materials you provide.
df <- data.frame( Run = c(1,2), Item = c(1,2), Event= c(1,1), Condition = c(1,2), TargetPrompt = c("1","2") ) ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt) Design=experimentDesign(ExperimentItem,session=1,randomItem=TRUE)
df <- data.frame( Run = c(1,2), Item = c(1,2), Event= c(1,1), Condition = c(1,2), TargetPrompt = c("1","2") ) ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt) Design=experimentDesign(ExperimentItem,session=1,randomItem=TRUE)
Prepares the stimuli data for the experiment.
loadData( runList, itemList, conditionList, promptList, header = TRUE, eventList = NULL )
loadData( runList, itemList, conditionList, promptList, header = TRUE, eventList = NULL )
runList |
A numeric vector of data representing the 'Run' column in the experiment. |
itemList |
A numeric vector of data representing the 'Item' column in the experiment. |
conditionList |
A numeric/character vector of data representing the 'Condition' column in the experiment. |
promptList |
A character vector of the main prompt (usually experiment items). |
header |
A logical value indicating if the output data.frame should include column headers (default is TRUE). |
eventList |
A numeric vector of data representing the 'Event' column in the experiment. |
A data frame with the processed columns 'Run', 'Event', 'Item', 'Condition', and 'Prompt', ready for use in experiments.
df <- data.frame( Run = c(1,2), Item = c(1,2), Condition = c(1,2), TargetPrompt = c("1","2"), Event = c(1,1) ) ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt)
df <- data.frame( Run = c(1,2), Item = c(1,2), Condition = c(1,2), TargetPrompt = c("1","2"), Event = c(1,1) ) ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt)
This function provides the number of tokens in a specified text, acting as a wrapper for an internal tokenizer function.
magicTokenizer(text)
magicTokenizer(text)
text |
A character string: the text for which the number of tokens is required. |
Returns the number of tokens in the provided text.
Configures experimental parameters before execution.
preCheck( data, checkToken = FALSE, systemPrompt = "", imgDetail = "auto", version = "2023-06-01", ... )
preCheck( data, checkToken = FALSE, systemPrompt = "", imgDetail = "auto", version = "2023-06-01", ... )
data |
A data.frame that has been structured by the 'experimentDesign' function, containing the experimental setup. |
checkToken |
Whether to perform token count check, select TRUE to submit your experiment to our server's tokenizer for token count check, the default selection is FALSE (i.e., no token check will be performed, but you need to manually check if the number of tokens exceeds the model limit to avoid errors in the experiment). |
systemPrompt |
The system prompt text used in the chatGPT model interaction. If left empty, a space character is assumed.Note: This parameter does not work in models that do not support system prompts. |
imgDetail |
The image quality of the img modality is set to auto by default, with low/high as selectable options. |
version |
When using the Claude model, the version parameter required defaults to "2023-06-01". |
... |
Variable parameter lists allow you to input additional parameters supported by the model you're using, such as n=2 / logprobs=TRUE... Note: You must ensure the validity of the parameters you enter; otherwise, an error will occur. |
A list containing the original data and the parameters for the chatGPT model interaction, confirming that the setup has passed the token checks or indicating issues if found.
df <- data.frame( Run = c(1,2), Item = c(1,2), Event =c(1,1), Condition = c(1,2), TargetPrompt = c("please repeat this sentence: test1","please repeat this sentence: test2") ) ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt) Design=experimentDesign(ExperimentItem,session=1) gptConfig=preCheck(Design, systemPrompt="You are a participant in a psychological experiment", imgDetail="low",temperature=0.7)
df <- data.frame( Run = c(1,2), Item = c(1,2), Event =c(1,1), Condition = c(1,2), TargetPrompt = c("please repeat this sentence: test1","please repeat this sentence: test2") ) ExperimentItem=loadData(df$Run,df$Item,df$Event,df$Condition,promptList = df$TargetPrompt) Design=experimentDesign(ExperimentItem,session=1) gptConfig=preCheck(Design, systemPrompt="You are a participant in a psychological experiment", imgDetail="low",temperature=0.7)
Executes the experiment and saves the results to an Excel file.
runExperiment(gptConfig, savePath = "./output.xlsx")
runExperiment(gptConfig, savePath = "./output.xlsx")
gptConfig |
A list containing the configuration for the language model, including the system prompt, model specifications, and token settings. |
savePath |
The file path where the experiment results will be saved in Excel format. Defaults to './output.xlsx' in the current working directory. |
This function does not return a value but saves the experiment results to the specified Excel file. Upon completion, "Done." will be printed to the console.
## Not run: runExperiment(Experiment_config,"./output.xlsx") #The first argument Experiment_config is generated by preCheck() function. Experiment_config <- preCheck(data) ## End(Not run)
## Not run: runExperiment(Experiment_config,"./output.xlsx") #The first argument Experiment_config is generated by preCheck() function. Experiment_config <- preCheck(data) ## End(Not run)
This function allows users to set and verify an API key for data collection. You can change the default api_url for others models' API.
setKey(api_key, model, api_url = NULL, ...)
setKey(api_key, model, api_url = NULL, ...)
api_key |
A character string: the user's OpenAI/huggingface/gemini/claude/baichuan/other API key. Please fill 'NA' for self-deployed models. |
model |
A character string: specify the model version. For gemini, you could input "gemini-pro" |
api_url |
A character string: the API URL for the model. If not specified, the default Chat completion URL will be used based on the api_key. |
... |
Additional arguments to be passed (currently not used, but kept for future extensibility). |
Prints a message to the console indicating whether the API key setup was successful. If the setup fails, the function stops with an error message.
## Not run: set_key(api_key="YOUR_API_KEY", model="gpt-3.5-turbo", api_url="api.openai.com/v1/chat/completions") ## End(Not run)
## Not run: set_key(api_key="YOUR_API_KEY", model="gpt-3.5-turbo", api_url="api.openai.com/v1/chat/completions") ## End(Not run)