{"id":51678,"date":"2025-03-29T14:50:15","date_gmt":"2025-03-29T09:20:15","guid":{"rendered":"http:\/\/officechai.com\/?p=51678"},"modified":"2025-03-29T14:50:16","modified_gmt":"2025-03-29T09:20:16","slug":"rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton","status":"publish","type":"post","link":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/","title":{"rendered":"RLHF Is Cr*p, It&#8217;s A Paint Job On A Rusty Car: Geoffrey Hinton"},"content":{"rendered":"\n<p>RLHF, or Reinforcement Learning from Human Feedback, is behind some of the recent advances in AI, but one of the pioneers of the field doesn&#8217;t think highly of it.<\/p>\n\n\n\n<p>Geoffrey Hinton, often called the &#8220;Godfather of AI,&#8221; has some harsh words for RLHF, comparing it to a shoddy paint job on a rusty car. &#8220;I think our RLHF is a pile of crap,&#8221; Hinton <a href=\"https:\/\/www.youtube.com\/watch?v=61BGgwN36Ps\">said <\/a>in a recent interview. &#8220;You design a huge piece of software that has gazillions of bugs in it. And then you say what I&#8217;m going to do is I&#8217;m going to go through and try and block each and put a finger in each hole in the dyke,&#8221; he said.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" fetchpriority=\"high\" decoding=\"async\" width=\"640\" height=\"336\" src=\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?resize=640%2C336\" alt=\"\" class=\"wp-image-49628\" srcset=\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?resize=1024%2C538&amp;ssl=1 1024w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?resize=300%2C158&amp;ssl=1 300w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?resize=768%2C403&amp;ssl=1 768w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?w=1200&amp;ssl=1 1200w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p>&#8220;It&#8217;s just no way. We know that&#8217;s not how you design software. You design it so you have some kind of guarantees. Suppose you have a car and it&#8217;s all full of little holes and rusty. And you want to sell it. What you do is you do a paint job. That&#8217;s what our RLHF is, a paint job,&#8221; he added.<\/p>\n\n\n\n<p>Reinforcement Learning from Human Feedback (RLHF) is a machine learning technique that helps build artificial intelligence (AI) models by integrating human feedback into the training process. Unlike traditional reinforcement learning, where AI agents learn through predefined reward functions, RLHF incorporates human preferences to refine the model&#8217;s behavior. This approach involves training a reward model using human evaluations of AI-generated outputs, which is then employed to optimize the AI&#8217;s decision-making. RLHF is particularly effective for complex tasks, such as natural language processing (NLP), where aligning AI outputs with nuanced human goals\u2014like humor or factual accuracy\u2014is challenging to define algorithmically. <\/p>\n\n\n\n<p>Hinton&#8217;s critique suggests that RLHF is a superficial fix, addressing surface-level issues without solving the underlying problems. He argues that instead of patching individual flaws in a fundamentally broken system, the focus should be on designing AI systems with inherent guarantees of safety and reliability. The &#8220;gazillions of bugs&#8221; he refers to might include biases, inaccuracies, and unpredictable behaviors that emerge from the complexity of these models. The &#8220;paint job&#8221; of RLHF, in his view, merely masks these flaws without addressing the underlying rot.<\/p>\n\n\n\n<p>This critique carries significant weight coming from someone of Hinton&#8217;s stature. He is one of the <a href=\"https:\/\/officechai.com\/stories\/how-ilya-sutskever-had-impressed-geoffrey-hinton-as-a-student\/\">pioneers <\/a>of Deep Learning, and won the Nobel Prize in 2024 for his contributions. Hinton&#8217;s critique highlights a growing concern within the AI community: that the current trajectory of AI development, while producing impressive results, might be building upon a shaky foundation. Meta&#8217;s AI Chief Yann LeCun has also been regularly <a href=\"https:\/\/officechai.com\/ai\/am-no-longer-interested-in-llms-they-are-just-token-generators-metas-ai-chief-yann-lecun\/\">saying <\/a>that current AI approaches will not necessarily be able to model human intelligence, and will eventually plateau in capabilities. It remains how AI progress unfolds, but there are several dissenting voices which say that current techniques won&#8217;t get us very far.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>RLHF, or Reinforcement Learning from Human Feedback, is behind some of the recent advances in AI, but one of the pioneers of the&#8230;<\/p>\n","protected":false},"author":1,"featured_media":49628,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1029],"tags":[],"class_list":["post-51678","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>RLHF Is Cr*p, It&#039;s A Paint Job On A Rusty Car: Geoffrey Hinton<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"RLHF Is Cr*p, It&#039;s A Paint Job On A Rusty Car: Geoffrey Hinton\" \/>\n<meta property=\"og:description\" content=\"RLHF, or Reinforcement Learning from Human Feedback, is behind some of the recent advances in AI, but one of the pioneers of the...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/\" \/>\n<meta property=\"og:site_name\" content=\"OfficeChai\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/OfficeChai\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-03-29T09:20:15+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-29T09:20:16+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"OfficeChai Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@OfficeChai\" \/>\n<meta name=\"twitter:site\" content=\"@OfficeChai\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"OfficeChai Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/\",\"url\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/\",\"name\":\"RLHF Is Cr*p, It's A Paint Job On A Rusty Car: Geoffrey Hinton\",\"isPartOf\":{\"@id\":\"https:\/\/officechai.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1\",\"datePublished\":\"2025-03-29T09:20:15+00:00\",\"dateModified\":\"2025-03-29T09:20:16+00:00\",\"author\":{\"@id\":\"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2\"},\"breadcrumb\":{\"@id\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1\",\"width\":1200,\"height\":630},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/officechai.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"RLHF Is Cr*p, It&#8217;s A Paint Job On A Rusty Car: Geoffrey Hinton\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/officechai.com\/#website\",\"url\":\"https:\/\/officechai.com\/\",\"name\":\"OfficeChai\",\"description\":\"Startups, Businesses And Careers\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/officechai.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2\",\"name\":\"OfficeChai Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/officechai.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g\",\"caption\":\"OfficeChai Team\"},\"description\":\"Dotting the i's, crossing the t's.\",\"url\":\"https:\/\/officechai.com\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"RLHF Is Cr*p, It's A Paint Job On A Rusty Car: Geoffrey Hinton","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/","og_locale":"en_US","og_type":"article","og_title":"RLHF Is Cr*p, It's A Paint Job On A Rusty Car: Geoffrey Hinton","og_description":"RLHF, or Reinforcement Learning from Human Feedback, is behind some of the recent advances in AI, but one of the pioneers of the...","og_url":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/","og_site_name":"OfficeChai","article_publisher":"https:\/\/www.facebook.com\/OfficeChai\/","article_published_time":"2025-03-29T09:20:15+00:00","article_modified_time":"2025-03-29T09:20:16+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1","type":"image\/jpeg"}],"author":"OfficeChai Team","twitter_card":"summary_large_image","twitter_creator":"@OfficeChai","twitter_site":"@OfficeChai","twitter_misc":{"Written by":"OfficeChai Team","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/","url":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/","name":"RLHF Is Cr*p, It's A Paint Job On A Rusty Car: Geoffrey Hinton","isPartOf":{"@id":"https:\/\/officechai.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#primaryimage"},"image":{"@id":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1","datePublished":"2025-03-29T09:20:15+00:00","dateModified":"2025-03-29T09:20:16+00:00","author":{"@id":"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2"},"breadcrumb":{"@id":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#primaryimage","url":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1","contentUrl":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1","width":1200,"height":630},{"@type":"BreadcrumbList","@id":"https:\/\/officechai.com\/ai\/rlhf-is-crp-its-a-paint-job-on-a-rusty-car-geoffrey-hinton\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/officechai.com\/"},{"@type":"ListItem","position":2,"name":"RLHF Is Cr*p, It&#8217;s A Paint Job On A Rusty Car: Geoffrey Hinton"}]},{"@type":"WebSite","@id":"https:\/\/officechai.com\/#website","url":"https:\/\/officechai.com\/","name":"OfficeChai","description":"Startups, Businesses And Careers","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/officechai.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2","name":"OfficeChai Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/officechai.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g","caption":"OfficeChai Team"},"description":"Dotting the i's, crossing the t's.","url":"https:\/\/officechai.com\/author\/admin\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2024\/10\/MixCollage-27-Oct-2024-08-52-PM-9488.jpg?fit=1200%2C630&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p685C6-drw","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts\/51678","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/comments?post=51678"}],"version-history":[{"count":1,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts\/51678\/revisions"}],"predecessor-version":[{"id":51682,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts\/51678\/revisions\/51682"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/media\/49628"}],"wp:attachment":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/media?parent=51678"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/categories?post=51678"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/tags?post=51678"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}