Large language models (LLMs) are used to generate content for a wide range of
tasks, and are set to reach a growing audience in coming years due to
integration in product interfaces like ChatGPT or search engines like Bing.
This intensifies the need to ensure that models are aligned with human
preferences and do not produce unsafe, inaccurate or toxic outputs. While
alignment techniques like reinforcement learning with human feedback (RLHF) and
red-teaming can mitigate some safety concerns and improve model capabilities,
it is unlikely that an aggregate fine-tuning process can adequately represent
the full range of users' preferences and values. Different people may
legitimately disagree on their preferences for language and conversational
norms, as well as on values or ideologies which guide their communication.
Personalising LLMs through micro-level preference learning processes may result
in models that are better aligned with each user. However, there are several
normative challenges in defining the bounds of a societally-acceptable and safe
degree of personalisation. In this paper, we ask how, and in what ways, LLMs
should be personalised. First, we review literature on current paradigms for
aligning LLMs with human feedback, and identify issues including (i) a lack of
clarity regarding what alignment means; (ii) a tendency of technology providers
to prescribe definitions of inherently subjective preferences and values; and
(iii) a 'tyranny of the crowdworker', exacerbated by a lack of documentation in
who we are really aligning to. Second, we present a taxonomy of benefits and
risks associated with personalised LLMs, for individuals and society at large.
Finally, we propose a three-tiered policy framework that allows users to
experience the benefits of personalised alignment, while restraining unsafe and
undesirable LLM-behaviours within (supra-)national and organisational bounds.
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.