{"id":1523,"date":"2026-06-26T04:00:00","date_gmt":"2026-06-26T04:00:00","guid":{"rendered":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/"},"modified":"2026-06-26T14:59:49","modified_gmt":"2026-06-26T14:59:49","slug":"2410-13077","status":"publish","type":"post","link":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/","title":{"rendered":"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble"},"content":{"rendered":"<p><br \/>\n<\/p>\n<div id=\"content-inner\">\n<div id=\"abs\">\n<div class=\"dateline\">\n  [Submitted on 16 Oct 2024 (v1), last revised 24 Jun 2026 (this version, v2)]<\/div>\n<p>View a PDF of the paper titled Tuning Language Fashions by Combination-of-Depths Ensemble, by Haoyan Luo and 1 different authors<\/p>\n<p>    View PDF<\/p>\n<blockquote class=\"abstract mathjax\"><p>\n            <span class=\"descriptor\">Summary:<\/span>Transformer-based Massive Language Fashions (LLMs) historically depend on final-layer loss for finetuning and final-layer representations for predictions, probably overlooking the predictive energy embedded in late layers. Interpretability instruments such because the logit lens present that late-layer representations already carry largely fashioned, task-relevant predictions; right here we ask whether or not that remark could be changed into an actionable coaching sign. We discover that focusing tuning effort on these layers can yield losses akin to these of the ultimate layer, with complementary test-time behaviour. Constructing on this, we introduce a tuning framework, Combination-of-Depths Ensemble (MoDE), which treats the late layers as an ensemble that contributes to the ultimate logits by discovered routing weights. MoDE could be utilized on high of any current tuning methodology (e.g., LoRA) and, in our experiments, modestly improves reasoning efficiency at a small parameter overhead. We current MoDE as a mechanism research displaying that late-layer logits could be made instantly helpful for tuning, and that they&#8217;ll substitute for considerably bigger trainable modules with comparable efficiency.\n    <\/p><\/blockquote><\/div>\n<\/div>\n<div>\n<h2>Submission historical past<\/h2>\n<p> From: Haoyan Luo [view email]                  [v1]<br \/>\n        Wed, 16 Oct 2024 22:51:45 UTC (12,478 KB)<br \/>\n    [v2]<br \/>\n        Wed, 24 Jun 2026 22:05:06 UTC (6,261 KB)\n<\/p><\/div>\n<p><br \/>\n<br \/><a href=\"https:\/\/arxiv.org\/abs\/2410.13077\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>[Submitted on 16 Oct 2024 (v1), last revised 24 Jun 2026 (this version, v2)] View a PDF of the paper titled Tuning Language Fashions by Combination-of-Depths Ensemble, by Haoyan Luo and 1 different authors View PDF Summary:Transformer-based Massive Language Fashions (LLMs) historically depend on final-layer loss for finetuning and final-layer representations for predictions, probably overlooking [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1525,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png","fifu_image_alt":"","jnews-multi-image_gallery":[],"jnews_single_post":[],"jnews_primary_category":[],"jnews_override_bookmark_settings":[],"jnews_social_meta":[],"jnews_override_counter":[],"footnotes":""},"categories":[2],"tags":[1977,50,1976,293,1074],"class_list":["post-1523","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-research-breakthroughs","tag-ensemble","tag-language","tag-mixtureofdepths","tag-models","tag-tuning"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble - Future News 24<\/title>\n<meta name=\"description\" content=\"Abstract page for arXiv paper 2410.13077: Tuning Language Models by Mixture-of-Depths Ensemble\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble - Future News 24\" \/>\n<meta property=\"og:description\" content=\"Abstract page for arXiv paper 2410.13077: Tuning Language Models by Mixture-of-Depths Ensemble\" \/>\n<meta property=\"og:url\" content=\"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/\" \/>\n<meta property=\"og:site_name\" content=\"Future News 24\" \/>\n<meta property=\"article:published_time\" content=\"2026-06-26T04:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-06-26T14:59:49+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png\" \/>\n<meta name=\"author\" content=\"Future News 24\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:image\" content=\"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Future News 24\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/\"},\"author\":{\"name\":\"Future News 24\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#\\\/schema\\\/person\\\/cecad1bde21cfc357cf70128144d6c83\"},\"headline\":\"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble\",\"datePublished\":\"2026-06-26T04:00:00+00:00\",\"dateModified\":\"2026-06-26T14:59:49+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/\"},\"wordCount\":230,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/arxiv.org\\\/static\\\/browse\\\/0.3.4\\\/images\\\/arxiv-logo-fb.png\",\"keywords\":[\"Ensemble\",\"Language\",\"MixtureofDepths\",\"Models\",\"Tuning\"],\"articleSection\":[\"AI Research &amp; Breakthroughs\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/\",\"url\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/\",\"name\":\"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble - Future News 24\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/arxiv.org\\\/static\\\/browse\\\/0.3.4\\\/images\\\/arxiv-logo-fb.png\",\"datePublished\":\"2026-06-26T04:00:00+00:00\",\"dateModified\":\"2026-06-26T14:59:49+00:00\",\"description\":\"Abstract page for arXiv paper 2410.13077: Tuning Language Models by Mixture-of-Depths Ensemble\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#primaryimage\",\"url\":\"http:\\\/\\\/arxiv.org\\\/static\\\/browse\\\/0.3.4\\\/images\\\/arxiv-logo-fb.png\",\"contentUrl\":\"http:\\\/\\\/arxiv.org\\\/static\\\/browse\\\/0.3.4\\\/images\\\/arxiv-logo-fb.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/2026\\\/06\\\/26\\\/2410-13077\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/futurenews24.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#website\",\"url\":\"https:\\\/\\\/futurenews24.com\\\/\",\"name\":\"Future News 24\",\"description\":\"The Smart Hub for AI and Next-Gen Innovation\",\"publisher\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/futurenews24.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#organization\",\"name\":\"Future News 24\",\"url\":\"https:\\\/\\\/futurenews24.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/futurenews24.com\\\/wp-content\\\/uploads\\\/2026\\\/06\\\/fn24-favicon.png\",\"contentUrl\":\"https:\\\/\\\/futurenews24.com\\\/wp-content\\\/uploads\\\/2026\\\/06\\\/fn24-favicon.png\",\"width\":250,\"height\":250,\"caption\":\"Future News 24\"},\"image\":{\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/futurenews24.com\\\/#\\\/schema\\\/person\\\/cecad1bde21cfc357cf70128144d6c83\",\"name\":\"Future News 24\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d57f07142d73cb5503ab2446ea7bc9ef3d0a5ba378d64a6157692311e42bf097?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d57f07142d73cb5503ab2446ea7bc9ef3d0a5ba378d64a6157692311e42bf097?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/d57f07142d73cb5503ab2446ea7bc9ef3d0a5ba378d64a6157692311e42bf097?s=96&d=mm&r=g\",\"caption\":\"Future News 24\"},\"sameAs\":[\"https:\\\/\\\/futurenews24.com\"],\"url\":\"https:\\\/\\\/futurenews24.com\\\/index.php\\\/author\\\/mridulpahuja20\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble - Future News 24","description":"Abstract page for arXiv paper 2410.13077: Tuning Language Models by Mixture-of-Depths Ensemble","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/","og_locale":"en_US","og_type":"article","og_title":"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble - Future News 24","og_description":"Abstract page for arXiv paper 2410.13077: Tuning Language Models by Mixture-of-Depths Ensemble","og_url":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/","og_site_name":"Future News 24","article_published_time":"2026-06-26T04:00:00+00:00","article_modified_time":"2026-06-26T14:59:49+00:00","og_image":[{"url":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png","type":"","width":"","height":""}],"author":"Future News 24","twitter_card":"summary_large_image","twitter_image":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png","twitter_misc":{"Written by":"Future News 24","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#article","isPartOf":{"@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/"},"author":{"name":"Future News 24","@id":"https:\/\/futurenews24.com\/#\/schema\/person\/cecad1bde21cfc357cf70128144d6c83"},"headline":"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble","datePublished":"2026-06-26T04:00:00+00:00","dateModified":"2026-06-26T14:59:49+00:00","mainEntityOfPage":{"@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/"},"wordCount":230,"commentCount":0,"publisher":{"@id":"https:\/\/futurenews24.com\/#organization"},"image":{"@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#primaryimage"},"thumbnailUrl":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png","keywords":["Ensemble","Language","MixtureofDepths","Models","Tuning"],"articleSection":["AI Research &amp; Breakthroughs"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/","url":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/","name":"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble - Future News 24","isPartOf":{"@id":"https:\/\/futurenews24.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#primaryimage"},"image":{"@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#primaryimage"},"thumbnailUrl":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png","datePublished":"2026-06-26T04:00:00+00:00","dateModified":"2026-06-26T14:59:49+00:00","description":"Abstract page for arXiv paper 2410.13077: Tuning Language Models by Mixture-of-Depths Ensemble","breadcrumb":{"@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#primaryimage","url":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png","contentUrl":"http:\/\/arxiv.org\/static\/browse\/0.3.4\/images\/arxiv-logo-fb.png"},{"@type":"BreadcrumbList","@id":"https:\/\/futurenews24.com\/index.php\/2026\/06\/26\/2410-13077\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/futurenews24.com\/"},{"@type":"ListItem","position":2,"name":"[2410.13077] Tuning Language Fashions by Combination-of-Depths Ensemble"}]},{"@type":"WebSite","@id":"https:\/\/futurenews24.com\/#website","url":"https:\/\/futurenews24.com\/","name":"Future News 24","description":"The Smart Hub for AI and Next-Gen Innovation","publisher":{"@id":"https:\/\/futurenews24.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/futurenews24.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/futurenews24.com\/#organization","name":"Future News 24","url":"https:\/\/futurenews24.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/futurenews24.com\/#\/schema\/logo\/image\/","url":"https:\/\/futurenews24.com\/wp-content\/uploads\/2026\/06\/fn24-favicon.png","contentUrl":"https:\/\/futurenews24.com\/wp-content\/uploads\/2026\/06\/fn24-favicon.png","width":250,"height":250,"caption":"Future News 24"},"image":{"@id":"https:\/\/futurenews24.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/futurenews24.com\/#\/schema\/person\/cecad1bde21cfc357cf70128144d6c83","name":"Future News 24","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/d57f07142d73cb5503ab2446ea7bc9ef3d0a5ba378d64a6157692311e42bf097?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/d57f07142d73cb5503ab2446ea7bc9ef3d0a5ba378d64a6157692311e42bf097?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d57f07142d73cb5503ab2446ea7bc9ef3d0a5ba378d64a6157692311e42bf097?s=96&d=mm&r=g","caption":"Future News 24"},"sameAs":["https:\/\/futurenews24.com"],"url":"https:\/\/futurenews24.com\/index.php\/author\/mridulpahuja20\/"}]}},"_links":{"self":[{"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/posts\/1523","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/comments?post=1523"}],"version-history":[{"count":1,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/posts\/1523\/revisions"}],"predecessor-version":[{"id":1524,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/posts\/1523\/revisions\/1524"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/media\/1525"}],"wp:attachment":[{"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/media?parent=1523"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/categories?post=1523"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/futurenews24.com\/index.php\/wp-json\/wp\/v2\/tags?post=1523"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}