Seedcorn 2020/2021

People who have communication difficulties find it challenging to express themselves effectively. People can experience difficulties in communication as a result of a range of conditions; some acquired in birth or early childhood, such as cerebral palsy, others as a result of a medical event or acquired condition that occurs in adulthood such as stroke, brain injury and motor neuron disease.

Approximately 300,000 people in the UK have complex communication difficulties that could benefit from external devices or communication aids. Communication aids, otherwise known as augmentative and alternative communication (AAC), refer to a set of tools or strategies that range from simple paper-based systems to electronic or digital technologies.

The main aim of traditional AAC systems is to convey a message or idea into a linguistic output via text, image or synthetic speech to facilitate communication. Robotics have also been used to support those with communication impairments, for example in the form of companion robots that engage people in social interactions and support play and interaction with peers.

Digital AAC technologies are often oriented to supporting verbal communication and ‘voice’. They also tend to support delivery of content or information from the person using the device, rather than facilitate richer social interaction or dialogue or allow for unscripted, spontaneous communication. Furthermore, many such technologies and robotics designed to support disabled people are developed alongside a medicalized perspective of disability, in which the technology is seen to ‘fix’ an impairment or help the person to become more ‘normal’. Some studies suggest that some communication aids can further perpetuate the structural inequalities or stigma associated with communication difficulties. Communication aid use is also deeply connected to the social and cultural context in which it is used, but that context is often overlooked or ignored.

What did the project involve? 

Softly non-spoken is a project about using soft robotics to develop non-verbal communication aids for people with communication difficulties. This multi-disciplinary team will bring together speech and language therapists, researchers in communication, and roboticists from academia and industry to investigate the use of current soft robotics research in a new, and
often overlooked, healthcare application. The aim is to provide better support than current devices, that focus on delivery of specific predetermined content only, by developing AAC that facilitates richer social interaction and allows for unscripted, spontaneous communication.

The field of soft robotics has the potential to provide superior alternative modes of communication to address some of these issues. Soft robots are constructed from materials similar to those found in living organisms and can also move and adapt to their surroundings like living things. As a result, soft robotic actuators are increasingly being used where organic motion is desired. This includes where non-verbal communication is the objective, for example in humanoid facial expression generation. Use of soft robotics principles to develop artificial intermediates that facilitate communication between humans is a current research theme within the Soft Lab at Bristol Robotics Laboratory.

This project moved away from technology as a way to fix or give ‘voice’ to people with communication difficulties and instead used previous EPSRC-funded research on Soft Robotics in a new human-robot interaction (HRI) context: to develop AAC devices that better understand and support interactions and communication in which people can express themselves in a variety of ways. Additionally, the work formed a new partnership between BRL and industry collaborators Air Giants with a shared goal of exploring the use of soft robots at different scales to develop methods of communicating that are known and valued in the world.

The project will drew on the diverse skill-set of a multidisciplinary team which included expertise in non-verbal communication and Soft Robotics. A key rationale for the project was the resulting knowledge exchange from this collaboration, in which those working in the fields of health, speech and language therapy, and education deepened an understanding of the potential of soft robotics to support user-controlled and directed communication and those designing and developing soft robotics technologies were able to consider new contexts and applications. Additionally, the project presented a unique opportunity to exploit the potential of soft robotics as a means to explore ways to better understand communication through movement, gesture, or touch, which may also enable more autonomous, unscripted ways of interacting.

The project had three fundamental aims:

Aim 1: To explore the use of soft robotics, in combination with wearable sensing, to develop soft-robotic AAC.

The objectives were to:

• Build on existing robot and sensor prototypes to create demonstrators that can be controlled manually, and later by different sensing modes (e.g. motion tracking, EMG, breathing sensor).
• Perform a comparative evaluation of the efficacy of different sensing modes for control of the robot by the person communicating.

Aim 2: To build prototypes that aim at conveying aspects of communication that are ignored by conventional assisted communication (e.g. body language, movement, gesture).

The objectives were to:

• Conduct an ideas generation phase with speech and language therapists and their wider community.
• Work with a puppeteer to devise expressive motion and feed this into the design of robot prototypes.
• Work with speech and language therapists and researchers in communication to perform comparative analysis against existing communication aids.

Aim 3: To gather feedback from public engagement activities with relevant stakeholder groups.

The objectives were to:

• Work with local AAC services, professionals and AAC users to share the project and identify useful networks to establish further involvement in the project and dissemination.
• Share the prototype with academic, health and charitable networks to encourage both discussion and further consideration of the use of robotics for the purpose of non-verbal communication as a way to raise awareness of this often-ignored communication issue.
• Establish a patient and public involvement group to provide feedback.

Who are the team and what do they bring?

• Hemma Philamore (Engineering Mathematics, University of Bristol) is a researcher with a focus in robotics and autonomous systems. She specializes in soft, energy-autonomous and bio-hybrid robots–including microbial and protein-based systems for electrical power generation and sensing.
• Alice Haynes (Engineering Mathematics, University of Bristol) is a maker and roboticist with a fascination with interfaces - the boundaries between us, within us and around us. How we interface with technology, other beings and ourselves. Alice Haynes' website.
• Katherine Broomfield (Bristol Speech and Language Therapy Research Unit, North Bristol NHS Trust) is a speech and language therapist. She is committed to helping people to communicate. She also wants to support people to be heard. She founded the Unspoken Voices Project.
• Richard Sewell (Air Giants) is a roboticist and at AirGiants he has worked on soft robotic technology and has experience with emotional communication through movement.
• Alison Oldfield (Education, University of Bristol) her research centres around learning and digital technologies, outdoor learning and inclusion. Previous research projects include the development of future classrooms across Europe, designing technologies for more inclusive classroom experiences, family learning and digital game-making, and young people's digital entrepreneurship.

What were the results?

Patient and public involvement in research: In anticipation of a user study with later stage prototypes, the project will make efforts to engage the specific community who are the intended end-users of this work - people whose main channel of communication is non-verbal. This will involve engaging partners such as Southmead Hospital to show demonstrators or videos conveying the work done in this investigation, within speech and language therapy units and establishing a patient and public involvement group to provide feedback. The team will work with local AAC services, professionals and AAC users to share the project and identify useful networks to establish further involvement in the project and dissemination.

Disseminate findings: The researchers will share the prototype with academic, health and charitable networks to encourage both discussion and further consideration of the use of robotics for the purpose of non-verbal communication. The findings of the study will be disseminated to the academic community via journal and/or conference papers in the areas of soft robotics and/or HRI. Furthering prototype development: They will take forward any positive outcomes and points for improvement of the devices from this exploratory stage and work towards a later stage, robotic AAC prototype that can ultimately be tested through a user study with people with communication difficulties (e.g. existing AAC users).

Further funding: The further device prototyping will be supported by follow-on funding applications. The researchers will employ the multi-disciplinarity of the team to target a wider range of funders than would typically be possible for a robotics project. Sources of funding may include the EPSRC IAA Follow-on Fund, and health funders such as NIHR, Research for Patient Benefit (RFPB) and i4i).