Cognitive representation and AI wellbeing

In a recent paper, Simon Goldstein and Cameron Domenico Kirk-Giannini argue that some current AI systems, namely artificial language agents, are plausible bearers of wellbeing. Central to their case is the claim that possession of mental states such as beliefs and desires is a precondition of wellbeing on several prominent accounts of wellbeing and that language agents possess such mental states. In this paper, I raise several issues for Goldstein and Kirk-Giannini’s case for language agent cognition and hence wellbeing. In particular, I argue that a careful look at both the behavioral outputs and the architecture of language agents raises doubts as to whether they possess mental states of the sort relevant to wellbeing. A broader lesson of my discussion is that work on AI wellbeing requires deeper engagement with philosophical and cognitive scientific work on the nature of states such as belief and desire.