logo
PolyU-led research reveals that sensory and motor inputs help large language models represent complex concepts

PolyU-led research reveals that sensory and motor inputs help large language models represent complex concepts

Malay Mail3 days ago

A research team led by Prof. Li Ping, Sin Wai Kin Foundation Professor in Humanities and Technology, Dean of the PolyU Faculty of Humanities and Associate Director of the PolyU-Hangzhou Technology and Innovation Research Institute, explored the similarities between large language models and human representations, shedding new light on the extent to which language alone can shape the formation and learning of complex conceptual knowledge.
HONG KONG SAR - Media OutReach Newswire - 9 June 2025 - Can one truly understand what "flower" means without smelling a rose, touching a daisy or walking through a field of wildflowers? This question is at the core of a rich debate in philosophy and cognitive science. While embodied cognition theorists argue that physical, sensory experience is essential to concept formation, studies of the rapidly evolving large language models (LLMs) suggest that language alone can build deep, meaningful representations of the world.By exploring the similarities between LLMs and human representations, researchers at The Hong Kong Polytechnic University (PolyU) and their collaborators have shed new light on the extent to which language alone can shape the formation and learning of complex conceptual knowledge. Their findings also revealed how the use of sensory input for grounding or embodiment – connecting abstract with concrete concepts during learning – affects the ability of LLMs to understand complex concepts and form human-like representations. The study, in collaboration with scholars from Ohio State University, Princeton University and City University of New York, was recently published in Nature Human Behaviour Led by Prof. LI Ping, Sin Wai Kin Foundation Professor in Humanities and Technology, Dean of the PolyU Faculty of Humanities and Associate Director of the PolyU-Hangzhou Technology and Innovation Research Institute, the research team selected conceptual word ratings produced by state-of-the-art LLMs, namely ChatGPT (GPT-3.5, GPT-4) and Google LLMs (PaLM and Gemini). They compared them with human-generated word ratings of around 4,500 words across non-sensorimotor (e.g., valence, concreteness, imageability), sensory (e.g., visual, olfactory, auditory) and motor domains (e.g., foot/leg, mouth/throat) from the highly reliable and validated Glasgow Norms and Lancaster Norms datasets.The research team first compared pairs of data from individual humans and individual LLM runs to discover the similarity between word ratings across each dimension in the three domains, using results from human-human pairs as the benchmark. This approach could, for instance, highlight to what extent humans and LLMs agree that certain concepts are more concrete than others. However, such analyses might overlook how multiple dimensions jointly contribute to the overall representation of a word. For example, the word pair "pasta" and "roses" might receive equally high olfactory ratings, but "pasta" is in fact more similar to "noodles" than to "roses" when considering appearance and taste. The team therefore conducted representational similarity analysis of each word as a vector along multiple attributes of non-sensorimotor, sensory and motor dimensions for a more complete comparison between humans and LLMs.The representational similarity analyses revealed that word representations produced by the LLMs were most similar to human representations in the non-sensorimotor domain, less similar for words in sensory domain and most dissimilar for words in motor domain. This highlights LLM limitations in fully capturing humans' conceptual understanding. Non-sensorimotor concepts are understood well but LLMs fall short when representing concepts involving sensory information like visual appearance and taste, and body movement. Motor concepts, which are less described in language and rely heavily on embodied experiences, are even more challenging to LLMs than sensory concepts like colour, which can be learned from textual data.In light of the findings, the researchers examined whether grounding would improve the LLMs' performance. They compared the performance of more grounded LLMs trained on both language and visual input (GPT-4, Gemini) with that of LLMs trained on language alone (GPT-3.5, PaLM). They discovered that the more grounded models incorporating visual input exhibited a much higher similarity with human representations.Prof. Li Ping said, "The availability of both LLMs trained on language alone and those trained on language and visual input, such as images and videos, provides a unique setting for research on how sensory input affects human conceptualisation. Our study exemplifies the potential benefits of multimodal learning, a human ability to simultaneously integrate information from multiple dimensions in the learning and formation of concepts and knowledge in general. Incorporating multimodal information processing in LLMs can potentially lead to a more human-like representation and more efficient human-like performance in LLMs in the future."Interestingly, this finding is also consistent with those of previous human studies indicating the representational transfer. Humans acquire object-shape knowledge through both visual and tactile experiences, with seeing and touching objects activating the same regions in human brains. The researchers pointed out that – as in humans – multimodal LLMs may use multiple types of input to merge or transfer representations embedded in a continuous, high-dimensional space. Prof. Li added, "The smooth, continuous structure of embedding space in LLMs may underlie our observation that knowledge derived from one modality could transfer to other related modalities. This could explain why congenitally blind and normally sighted people can have similar representations in some areas. Current limits in LLMs are clear in this respect".Ultimately, the researchers envision a future in which LLMs are equipped with grounded sensory input, for example, through humanoid robotics, allowing them to actively interpret the physical world and act accordingly. Prof. Li said, "These advances may enable LLMs to fully capture embodied representations that mirror the complexity and richness of human cognition, and a rose in LLM's representation will then be indistinguishable from that of humans."Hashtag: #PolyU #HumanCognition #LargeLanguageModels #LLMs #GenerativeAI
The issuer is solely responsible for the content of this announcement.

Orange background

Try Our AI Features

Explore what Daily8 AI can do for you:

Comments

No comments yet...

Related Articles

DayOne secures US$3.5bil financing for Johor data centres
DayOne secures US$3.5bil financing for Johor data centres

Free Malaysia Today

time11 hours ago

  • Free Malaysia Today

DayOne secures US$3.5bil financing for Johor data centres

Johor has about 30 data centre projects completed or under construction, and another 20 awaiting approval. (AP pic) PETALING JAYA : DayOne Data Centers Singapore Pte Ltd secured RM15 billion (US$3.5 billion) of multicurrency financing to support its green data centres in Malaysia's Johor state, according to a statement from Oversea-Chinese Banking Corporation Ltd (OCBC). 'OCBC and its Malaysian unit, OCBC Bank (Malaysia) Bhd acted as joint coordinators for the syndicated financing, which involved a RM7.5 billion Islamic financing and a US$1.7 billion offshore term-loan facility,' the bank said. 'The proceeds will be used for both refinancing and capital expenditure of DayOne's data centres, which are expected to obtain green digital infrastructure certification,' it said. Malaysia is turning into a beneficiary of Asia's growing need for data centres to support the boom in artificial intelligence. Johor, which sits across from the city-state of Singapore, has about 30 projects completed or under construction, plus 20 more awaiting approvals. Companies investing have included Microsoft Corp and ByteDance Ltd. DayOne, formerly known as GDS International, is the international arm of Chinese data centre operator GDS Holdings Ltd. The facility, which is its biggest-ever loan, would be among the largest syndicated financings for the sector by a borrower in Asia, Bloomberg News reported.

Hong Kong Hosts World's Largest ISSCR Stem Cell Gathering for the First Time
Hong Kong Hosts World's Largest ISSCR Stem Cell Gathering for the First Time

Malay Mail

time15 hours ago

  • Malay Mail

Hong Kong Hosts World's Largest ISSCR Stem Cell Gathering for the First Time

Some 2,400 stem cell and regenerative medicine pioneers convene to chart new frontiers, resulting from bidding efforts of Hong Kong Convention Ambassadors of the Hong Kong Tourism Board The University of Hong Kong, The Chinese University of Hong Kong and the Hong Kong University of Science and Technology, which are appointed organisations under the Hong Kong Convention Ambassador (HKCA) programme of the Hong Kong Tourism Board (HKTB). HONG KONG SAR - Media OutReach Newswire - 11 June 2025 - Reputed to be a leading research and development (R&D) hub for medical sciences, Hong Kong hosts the world's largest and most significant gathering of stem cell research and regenerative medicine, the International Society of Stem Cell Research (ISSCR) 2025 Annual Meeting. The event is brought to Hong Kong for the first time, thanks to the joint effort of three local universities –, which are appointed organisations under the Hong Kong Convention Ambassador (HKCA) programme of the Hong Kong Tourism Board (HKTB). The four-day meeting held on June 11-14 gathers more than 2,400 local and international experts in stem cell research and regenerative medicine from 58 countries and regions, from Asia to North America, Europe, Australia, the Middle East, and Mainland China. Pioneers share the latest advances in the field, with topics spanning the field of regenerative medicine including topics such as advances in cell therapies, fundamental discoveries, understanding disease processes and more. The four-day meeting held on June 11-14 gathers more than 2,400 local and international experts in stem cell research and regenerative medicine from 58 countries and regions, from Asia to North America, Europe, Australia, the Middle East, and Mainland China. Pioneers share the latest advances in the field, with topics spanning the field of regenerative medicine including topics such as advances in cell therapies, fundamental discoveries, understanding disease processes and more. Dr Peter Lam, HKTB Chairman, said, "The International Society for Stem Cell Research (ISSCR) is a globally leading organisation in medical sciences and research, and its annual meeting is one of the world's most significant international academic conventions in medical science. Hong Kong is the first city chosen to host this annual event in Asia after a 13-year hiatus, underscoring the city's status as the 'World's Meeting Place'. I am grateful for the active participation of the Hong Kong Convention Ambassadors from the three local universities, who leveraged their influence and network during the bidding for the host of the event to make the event come true. We are also delighted to see that more professionals from a wider range of expertise are joining HKTB's league of Convention Ambassadors, and we believe their participation can help identify and contribute to the staging of more significant international events in Hong Kong." Valentina Greco, ISSCR President, remarked, "Hosting ISSCR 2025 in Hong Kong marks a defining moment for global stem cell science. As a vibrant research hub at the crossroads of Asia and the world, Hong Kong offers an ideal setting to connect diverse scientific communities, spark new collaborations, and accelerate innovation. The ISSCR is deeply grateful to the city of Hong Kong and to our generous co-sponsors – the University of Hong Kong, The Chinese University of Hong Kong, and the Hong Kong University of Science and Technology – for their support and commitment to bringing The Global Stem Cell Event to this beautiful city. With the unwavering support of the Hong Kong Tourism Board, these three leading universities, and inspired leadership from our 2025 Annual Meeting Program Committee, this meeting exemplifies the power of partnership and shared vision. We are proud to return to Asia for the first time in more than a decade at a time when our collective progress in stem cell research and regenerative medicine and to advancing therapies that have life altering impact on patients depends on the exchange of bold ideas across borders." The extensive programme features plenary sessions, concurrent tracks, poster presentations, and networking opportunities. Contributing to the success of the event, the HKTB has provided comprehensive support, from arranging reconnaissance to venue sourcing, bid proposal, network support and more. Hong Kong Convention Ambassador Programme Aligns a Record Lineup of 180 Industry Leaders HKTB hosted the "Hong Kong Conference Ambassador Appreciation Night" last year to honor the contributions of the ambassadors in reinforcing Hong Kong as a 'World Meeting Place' Hong Kong Convention Ambassadors from the Mainland recently participated in a business forum. HKCAs have been instrumental in winning the ISSCR 2025 Annual Meeting for Hong Kong. Launched in 2020 by the HKTB as a strategic initiative, the HKCA programme recruits high-level representatives of Hong Kong's eight universities, chambers of commerce, professional organisations, public bodies, industrial entities and professional societies to act as ambassadors for Hong Kong's convention sector. With their professional influence and networks, HKCAs have led efforts to bring in events ranging from medical sciences to innovation and technology (I&T), financial services, aviation, transportation and more. The close collaboration between HKCAs and the HKTB has proven effective in promoting Hong Kong as a premier hub for world-class conventions. HKCAs have been instrumental in winning the ISSCR 2025 Annual Meeting for Hong Kong. Launched in 2020 by the HKTB as a strategic initiative, the HKCA programme recruits high-level representatives of Hong Kong's eight universities, chambers of commerce, professional organisations, public bodies, industrial entities and professional societies to act as ambassadors for Hong Kong's convention sector. With their professional influence and networks, HKCAs have led efforts to bring in events ranging from medical sciences to innovation and technology (I&T), financial services, aviation, transportation and more. The close collaboration between HKCAs and the HKTB has proven effective in promoting Hong Kong as a premier hub for world-class conventions. As an alliance of distinguished local and Mainland leaders in business, medicine, professional services, I&T and other fields, the HKCA programme has been growing from strength to strength, with 72 new members in the latest record-sized cohort for the 2025-2026 term, taking the total to about 180, some 30% more than the previous cohort. Among the 76 international events for 2025 to 2027 lined up by the HKTB, with an estimated attendance of over 180,000 MICE visitor arrivals and 27 resulted from HKCAs' efforts. The events cover diverse fields ranging from I&T to aviation and medical sciences. Some are held in Hong Kong for the first time, including IEEE Electron Devices Technology and Manufacturing Conference held in March 2025, Routes World in September 2025 and World Cancer Congress 2026. The HKTB is committed to working closely with HKCAs to welcome more world-class events to Hong Kong, attracting high-yield overnight business visitors to the city and reinforcing the city's status as the world's meeting place. Large-scale conventions facilitated by Hong Kong Convention Ambassadors: Date Events (*first-ever in Hong Kong) Theme/Sector March 2025 IEEE Electron Devices Technology and Manufacturing Conference * Innovation & Technology May 2025 IEEE International Conference on Data Engineering Innovation & Technology Jun 2025 International Society for Stem Cell Research (ISSCR) Annual Meeting* Medical sciences Sep 2025 Routes World* Aviation Dec 2025 The International Society of Sport Psychology 16th World Congress Sports & Medical sciences 2026 Asia-Pacific Association for International Education 2026 Conference and Exhibition Education 2026 17th Asian Congress of Oral and Maxillofacial Surgery Medical sciences 2026 Asian Venture Philanthropy Network Conference Philanthropy 2026 International Federation of Landscape Architects World Congress Architecture 2026 Lions International Convention Social Services 2026 The Association of National Olympic Committees General Assembly Sports 2026 The Council Meeting of the Asian Patent Attorneys Association Intellectual property 2026 World Cancer Congress* Medical sciences 2027 International Association for Energy Economics Annual International Conference Economics 2027 Lions Clubs International Orient and Southeast Asian Lions Forum Social Services 2027 World Organization of Family Doctors Asia Pacific Regional Conference Medical sciences Hashtag: #HKTB The issuer is solely responsible for the content of this announcement.

‘Malaysia is open for innovation': Gobind woos UK tech investors in London
‘Malaysia is open for innovation': Gobind woos UK tech investors in London

Malay Mail

time19 hours ago

  • Malay Mail

‘Malaysia is open for innovation': Gobind woos UK tech investors in London

KUALA LUMPUR, June 11 — Malaysia has called on the United Kingdom (UK) technology businesses to join hands in building a dynamic digital future in the country. Digital Minister Gobind Singh Deo, speaking at the UK-Malaysia Digital Gateway Forum in London, also positioned Malaysia as a regional powerhouse for innovation, digital investment, and sustainable technology development. 'Malaysia is open for innovation. Our vision is anchored on three critical enablers—world-class infrastructure, trusted data governance, and institutional trust. 'These are the foundations that make Malaysia a prime destination for high-value tech investment in the region,' he said in a statement today. Gobind is currently in London for London Tech Week, which also serves as a valuable platform for Malaysia to gain insights and forge partnerships in emerging areas such as artificial intelligence (AI) governance, smart cities, and digital sustainability. At the forum, he reiterated Malaysia's commitment to becoming a thriving, future-ready digital hub that welcomes international partnerships and is poised for high-value, sustainable growth. He cited Malaysia's strong digital growth trajectory, highlighting robust investment figures from the first quarter of 2025. 'During this period, 107 new companies were approved under the Malaysia Digital framework, representing a combined investment value of RM13.1 billion (GBP2.3 billion). 'These investments are expected to generate 4,199 new jobs over the next five years,' he said. Gobind also extended an invitation to UK partners to participate in Malaysia's upcoming flagship events, including the Asean Malaysia AI Summit scheduled for August 2025, and the Smart City Expo Kuala Lumpur, to be held in September 2025. — Bernama

DOWNLOAD THE APP

Get Started Now: Download the App

Ready to dive into the world of global news and events? Download our app today from your preferred app store and start exploring.
app-storeplay-store