Task Adapters Generation from Instructions (2024)

Huanxuan Liao^1,2, Shizhu He^1,2 , Yao Xu^1,2, Yuanzhe Zhang^1,2,
Yanchao Hao³, Shengping Liu⁴, Kang Liu^1,2, Jun Zhao^1,2
¹ The Key Laboratory of Cognition and Decision Intelligence for Complex Systems,
Institute of Automation, Chinese Academy of Sciences, Beijing, China
² School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
³ Platform and Content Group, Tencent, Beijing, China ⁴ Unisound, Beijing, China
liaohuanxuan2023@ia.ac.cn {shizhu.he, yao.xu, kliu, jzhao}@nlpr.ia.ac.cn
Corresponding author

Abstract

Large language models (LLMs) have acquired the ability to solve general tasks by utilizing instruction finetuning (IFT).However, IFT still relies heavily on instance training of extensive task data, which greatly limits the adaptability of LLMs to real-world scenarios where labeled task instances are scarce and broader task generalization becomes paramount.Contrary to LLMs, humans acquire skills and complete tasks not merely through repeated practice but also by understanding and following instructional guidelines. This paper is dedicated to simulating human learning to address the shortcomings of instance training, focusing on instruction learning to enhance cross-task generalization. Within this context, we introduce Task Adapters Generation from Instructions (TAGI), which automatically constructs the task-specific model in a parameter generation manner based on the given task instructions without retraining for unseen tasks.Specifically, we utilize knowledge distillation to enhance the consistency between TAGI developed through Learning with Instruction and task-specific models developed through Training with Instance, by aligning the labels, output logits, and adapter parameters between them. TAGI is endowed with cross-task generalization capabilities through a two-stage training process that includes hypernetwork pretraining and finetuning.We evaluate TAGI on the Super-Natural Instructions and P3 datasets. The experimental results demonstrate that TAGI can match or even outperform traditional meta-trained models and other hypernetwork models, while significantly reducing computational requirements. Our code will be available at https://github.com/Xnhyacinth/TAGI.

1 Introduction

Large language models (LLMs) have acquired the ability to solve general tasks by utilizing instruction finetuning (IFT), which describes different tasks in the same natural language format [3; 6; 22]. However, IFT still relies heavily on instance training of extensive task data {(Description, [Demostrations], Source, Target)} [36; 38], which faces significant limitations in adapting LLMs to real-world scenarios where labeled task instances are scarce and broader task generalization becomes paramount.

Therefore, for better cross-task generalization, the "zero-shot" learning ability of LLMs is crucial for real-world applications: models learned with instructions can achieve non-trivial performance on unseen tasks with just a single instruction that provides a comprehensive description of the task(e.g., "You will be given sentences in which your task is to recognize the name of a person."). Traditionally, achieving this capability involves meta-training the model by associating each input with specific task instructions [20; 36].For example, GPT-3 [24] has demonstrated strong "zero-shot" capabilities through meta-training. However, these methods heavily depend on the foundation model’s abilities and are inefficient for various unseen tasks [21; 43], as they require reprocessing extensive task instructions and some supplementary task data (e.g., examples from few-shot instances) for each input (see the top of Figure1).

Task Adapters Generation from Instructions (1)

In recent years, researchers have begun to explore meta-learning to enhance the cross-task generalization capabilities of LLMs, aiming to construct flexible, reusable and robust task-specific models [1; 33]. For example, task-specific models such as Adapter [11], LoRA [12], and Prefix [17] have been constructed by a hypernetwork [8]. This approach significantly enhances task generalization by processing instructions efficiently, reducing redundant computations [26]. However, these methods heavily depend on a substantial corpus of training instances, which can hinder their capacity to efficiently learn and construct task-specific models based on provided instructions [13].

In fact, contrary to LLMs, humans acquire skills and complete tasks not only through repeated practice but also by understanding and following instructional guidelines [15]. For example, a tourist with basic knowledge of riding vehicles can easily learn to use new ones abroad for the first time with the help of travel guides.This paper aims to mimic the way humans learn skills by understanding instructions. This shift represents a modest evolution in task model construction, transitioning from traditional instance training models to a contemporary approach focused on instruction learning.By providing task instructions, the novel paradigm offers an automated solution for generating task-specific adapters and seamlessly integrating them into the base model. This approach aims to streamline the development of task-specific models while enhancing their ability to generalize across diverse tasks with instructions.

Guided by this goal, we introduce Task Adapters Generation from Instructions (TAGI), which converts instructions to task-specific adapters using a hypernetwork. Under the knowledge distillation framework [10; 35], we enable models to the "Learning with Instruction" paradigm in a manner analogous to the "Training with Instance" paradigm. TAGI will enhance the alignment between the task-specific model $\theta_{k}$ (acting as the teacher) and the vanilla LLM $\theta_{0}$ combined with the generated task adapters $\Delta_{k}$ (acting as the student) (see the bottom of Figure1). This alignment is achieved not only through instance training but also by incorporating parameter learning for task-specific models based on instructions. Specifically, we align the student under two distinct paradigms, encompassing not just the targets and logits, but also the adapters’ parameters by an L2 regularization within instruction, which represents the enhancement of the understanding of instructions and the ability to generate more efficient task-specific adapters. Moreover, TAGI endows the model with task generalization capabilities through a two-stage training process: hypernetwork pretraining on standard text pretraining data (e.g., C4 [28]), followed by finetuning on meta-training tasks. This allows it to generalize effectively across unseen tasks without sacrificing performance.

We evaluate TAGI on the Super-Natural Instructions (SNI) [36] and P3 [29] datasets. Experimental results demonstrate its ability to effectively generate adapters for unseen tasks, surpassing meta-trained models by 2% in SNI and 5% in P3, while significantly reducing computational demands by 60%, and outperforming other hypernetwork models by 7%.Notably, our method does not require additional parameter updating or gradient back-propagation, and it avoids the inefficiency of repeatedly encoding instructions during inference.We summarize our contributions as follows:

•
We propose a novel model construction paradigm by imitating human learning abilities, Learning with Instruction, for the cross-task generalization of the LLMs. To the best of our knowledge, it is the first time that a task-specific model has been generated based on instruction learning, and its capabilities and parameters are distilled from a teacher model trained on instance learning.
•
We used a knowledge distillation framework to develop task-specific models within the instruction learning paradigm. By aligning model parameters comprehensively, the TAGI method improves the model’s ability to understand instructions and solve unseen tasks more accurately and efficiently.
•
Comprehensive quantitative and qualitative assessments have highlighted the effectiveness of TAGI on two publicly available large-scale instruction datasets, with lower inference costs.

2 Related Work

TAGI draws inspiration from previous research on instruction following, hypernetworks and knowledge distillation. In this section, we will delve into the pioneering work in these areas.

Instruction Following is often used to evaluate the cross-task generalization of LLMs, and it is dedicated to handling any task described in natural language. Recent findings suggest that additional finetuning of LLMs with instructions substantially improves their zero-shot capabilities [6; 37; 38].Moreover, large-scale multi-task meta-training has been shown to equip models with the ability to address new tasks in zero- or few-shot scenarios, facilitated by standard task formats and prompts [29; 43] alongside providing concise task instructions and select examples [23; 36]. However, the instructions and examples can significantly escalate the computational burden compared to task-specific models. Existing works attempt to mitigate this issue involved creating adapters to separately process instructions and examples [13; 41] with reduced performance. To overcome these limitations, we introduce a new paradigm that draws on instruction-based learning, simulating instance training to enhance the perception and processing capabilities of LLMs for handling unseen tasks.

Hypernetworks [8; 30] are neural networks that generate weights for other neural networks [4], which are designed to use fewer parameters to dynamically build task-specific models [9; 32]. Notable works such as HyperTuning [26], HINT [13], and Hypter [41] have all adopted hypernetworks to convert task instructions and demonstrations into adapters for LLMs. And MEND [5] utilizes hypernetworks to compress demonstrations for distilled vectors. Although they all avoided processing lengthy instructions repeatedly and utilized adapters to make training and testing more cost-effective [18], they still have a performance loss compared to meta-training [7]. The proposed method TAGI incorporates the utilization of hypernetworks, which are instrumental in generating task-specific adapters that are seamlessly integrated into LLMs. Compared to existing models based on hypernetworks, TAGI not only trains at the instance level but also incorporates knowledge distillation to supervise the adapters generated by hypernetworks, thereby achieving both efficiency and effectiveness.

Knowledge Distillation is a technique where a smaller model (student) learns to mimic the predictions of a larger model (teacher), aiming to retain performance while reducing computational resources [10]. Indeed, the application of knowledge distillation is the essential difference between the proposed method in this paper and other hypernetwork-based methods such as HINT [13] and Hypter [41]. Recently, some works [31] utilize knowledge distillation to finetune small language models such as T5 [28], enabling them to act as LLMs with pre-prompting without any given prompts. Compared with the typical knowledge distillation methods of LLMs, the proposed method TAGI in this paper further utilizes model parameter alignment and aims to mimic another learning paradigm of human skill learning. We not only calculate the Kullback–Leibler (KL) divergence [14] between teacher and student models [10], but also compute the L2 regularization between the generated adapter by instruction learning and task-specific models by instance training.

3 Methods

3.1 Problem Setting

Cross-task Generalization: Given a set of tasks $\mathcal{T}=\{\mathcal{T}_{1},...,\mathcal{T}_{|\mathcal{T}|}\}$ , where each task $\mathcal{T}_{i}$ contains a set of (source, target) samples $\mathcal{D}_{i}=\{(s_{1},t_{1}),...,(s_{n},t_{n})\}$ . We categorize these tasks into three distinct non-overlapping groups for validating out-of-distribution generalization: meta-train ( $\mathcal{T}_{train}$ ), meta-valid ( $\mathcal{T}_{valid}$ ), and meta-test ( $\mathcal{T}_{test}$ ), assuming all tasks adhere to a text-to-text format. For example, $\mathcal{T}_{train}$ comprises tasks like translation and question answering, the $\mathcal{T}_{valid}$ and $\mathcal{T}_{test}$ encompass tasks such as paraphrasing and natural language inference respectively.Within the $\mathcal{T}_{train}$ , the goal is to utilize the data for training and transfer knowledge to facilitate learning to resolve the test tasks. For all methods discussed, aside from the original unsupervised pretraining of the language model backbone on separate corpora, the model learning primarily takes place through multi-task training on the $\mathcal{T}_{train}$ .

3.2 Task Adapters Generation from Instructions (TAGI)

In this section, we will introduce the detailed method of TAGI. For each (unseen) task, TAGI consists of two core components: a hypernetwork §3.2.1 which receives task instructions and generates parameter-efficient adapters, and a task-specific model which combines the vanilla LLM and the generated adapters from hypernetwork.

Unlike traditional meta-training methods, we transition from training with instance to learning with instruction, which not only addresses efficiency issues at the instance level but also incorporates parameter alignment for the task-specific model parameters at the instruction level.Specifically, the complete process is shown in Figure2, we initially train the LoRA modules §3.2.2 on various upstream tasks (seen tasks) with task datasets of meta-train ( $\mathcal{T}_{train}$ ). Specifically, for $N$ distinct upstream tasks, we independently train $N$ LoRA modules, with each module denoted as $\Delta_{i}$ for task $\mathcal{T}_{i}\in\mathcal{T}$ , presumed to represent the optimal model for its respective task. Subsequently, TAGI is committed to building proprietary models for downstream tasks (unseen tasks). Its training process is bifurcated into two primary phases: hypernetwork pretraining §3.2.3 and hypernetwork finetuning §3.2.4 which encompasses distillation and alignment.

Task Adapters Generation from Instructions (2)

3.2.1 Hypernetwork for Converting Instructions into LoRA

A pivotal element of our model is the hypernetwork that converts task instructions (descriptions and demonstrations) into a parameter-efficient module. Our hypernetwork comprises two crucial components: the encoder, derived from the vanilla LLM¹¹1We find that re-using the encoder from the vanilla LLM works well [13]., is designed to minimize encoding biases by converting task instructions into a continuous contextual representation. This representation is then fused with LLM input and concated with encoded input for the decoder. Additionally, the adapter generator, utilizing a basic MLP design, is both lightweight and efficient, effectively converting encoded instructions into parameter-efficient modules.

Encoder: Prior studies simply concatenated encoded instructions with inputs, overlooking the interactions between them. To address this, we integrated a hierarchical cross-attention layer into the encoder of the LLM to refine the input representation with embedded instruction details. Specifically, for an input $x$ and its corresponding task instruction $i_{x}$ , we initially employ the encoder within the hypernetwork to encode the instruction into representations $\textbf{I}_{x}\in\mathbb{R}^{s\times d}$ . Then, we feed the $x$ into the model and obtain the output representation $\textbf{S}_{l}$ from the self-attention sublayer in the $l$ -th layer. Ultimately, $\textbf{S}_{l}$ is processed through the $l$ -th cross-attention layer, resulting in a text representation that is enriched with instruction information:

\textbf{F}_{l}=\text{CrossAttentionLayer}_{l}(\textbf{S}_{l},\textbf{I}_{x})

(1)

where $\text{CrossAttentionLayer}_{l}$ conducts multi-head attention on the query, key, and value matrices, followed by residual connection and layer normalization. The final input to the decoder is the concatenation of the encoded instruction and the encoded fusion input, i.e., $(\textbf{I}_{x};\textbf{F}_{l})$ .

Adapter Generator: Considering the efficiency and effectiveness, we utilize a two-layer multi-layer perceptron (MLP) to generate parameter-efficient modules (e.g., LoRA) for the encoded instruction. To differentiate between the query $\mathcal{Q}$ and value $\mathcal{V}$ matrices as well as the layers, we introduce layer ids $\text{idx}_{l}^{\{\mathcal{Q},\mathcal{V}\}}\in\{0,\ldots,2\times\#\text{%blocks}\}$ as positional information. We use a unique network for each layer and share it between $\mathcal{Q}$ and $\mathcal{V}$ (i.e., one network is used for a certain layer LoRA generation).

3.2.2 LoRA Tuning for Task-specific Models

LoRA [12] efficiently reduces the number of trainable parameters by decomposing the update of the LLM’s attention weight matrix (denoted as $\mathbf{W}_{0}\in\mathbb{R}^{d\times k}$ ) into low-rank matrices. Specifically, LoRA updates the weight matrix as $\mathbf{W}_{0}+\delta\mathbf{W}=\mathbf{W}_{0}+\mathbf{AB}$ , with $\mathbf{A}\in\mathbb{R}^{d\times r}$ and $\mathbf{B}\in\mathbb{R}^{r\times k}$ being trainable low-rank matrices of rank $r$ , significantly smaller in dimensions than $d$ and $k$ . We finetune a robust baseline to derive the LoRA parameters $\Delta_{i}$ for task-specific models for $i$ -th task, facilitating LLM instruction learning and parameter alignment. SNI is categorized into 60 types based on task types, while P3 encompasses 36 categories, corresponding to 60 and 36 parameter modules, respectively.

3.2.3 Hypernetwork Pretraining for Preliminary Generalization

Previous research [5; 26] has demonstrated that pretraining hypernetworks can substantially improve the model’s cross-task generalization capabilities. Adhering to the HINT [13], we pretrain the hypernetwork on C4 [28] before finetuning it on a diverse multi-task prompt dataset. As illustrated in the right segment of Figure2, given an input sequence, we partition it into randomly sized segments $a$ , $b$ , and $c$ , where $a$ is fed into the hypernetwork, $b$ into the LLM, and $c$ is the segment to predict. During this stage, training is conducted by minimizing the cross-entropy loss $\mathcal{L}_{\text{pred}}$ , aiming to ensure that the hypernetwork learns to recognize instructions to enhance generalization ability.

\mathcal{L}_{\text{pred}}=\text{log}P_{(\text{LLM}+\text{Hypernetwork}(a))}(c~%{}|~{}b)

(3)

3.2.4 Hypernetwork Finetuning for Instruction Learning

At this stage, TAGI is finetuned on a multi-task prompt dataset, enabling it to learn the generation of optimal parameters from task instructions, thereby ensuring effective generalization to future unseen tasks. Similar to the pretraining phase, task instructions (alongside some few-shot samples) replace $a$ , the main input replaces $b$ , and the target replaces $c$ . In each iteration, the hypernetwork generates LoRA parameters and encodes the instructions. LoRA is a parameter-efficient module (i.e., inserting into the model), and the encoded instructions are integrated with the encoder’s embeddings for information fusion and concatenated with the fused encoding input during decoding. Beyond the standard $\mathcal{L}_{\text{pred}}$ , we employ knowledge distillation for instruction learning: a strong baseline combining complete task instructions and input, serves as the teacher, while the model incorporating generated LoRA parameters with the input, acts as the student. The KL divergence $\mathcal{L}_{\text{kl}}$ measures the discrepancy in word probability distributions between the two models as an implicit learning outcome, and the MSE loss $\mathcal{L}_{\text{ins}}$ calculates the difference between the generated parameters and those of task-specific parameter-efficient modules as an explicit learning intermediate result. The formulation of finetuning is as follows:

\mathcal{L}_{\text{ins}}=\text{MSE}(\Delta_{i},\text{Hypernetwork}(a))

(4)

\mathcal{L}_{\text{kl}}=\text{KL}(P_{(\text{LLM}+\Delta_{i})}(x~{}|~{}(a;b))~{%}||~{}P_{(\text{LLM}+\text{Hypernetwork}(a))}(x~{}|~{}b))

(5)

\mathcal{L}_{\text{finetune}}=\mathcal{L}_{\text{pred}}+\lambda_{1}\mathcal{L}%_{\text{kl}}+\lambda_{2}\mathcal{L}_{\text{ins}}

(6)

where $a\in\mathcal{T}_{i}$ , $\Delta_{i}$ is the optimal LoRA modules of the $i$ -th task, $\lambda_{1}$ and $\lambda_{2}$ are the hyper-parameter to control the importance of distillation in finetuning.

4 Experiments

We first present the datasets (§4.1) and baselines (§4.2) used in our evaluation and then discuss three research questions (RQs):

RQ1: Can the proposed instruction learning paradigm effectively learn the ability of instance training? Can it support cross-task generalization of LLMs? (§4.4)

RQ2: How many foundation tasks does TAGI need to learn to achieve better results? (§4.5)

RQ3: What is the impact of different modules and learning stages on TAGI? (§4.7)

4.1 Datasets

To demonstrate the generality of our method, we evaluate our approach on two popular multi-task instruction datasets²²2We provide the full list of datasets and more details in the A.2.: Super-Natural Instructions (SNI) [36] and T0 split of P3 (P3) [29].

SNI comprising over 1,600 task datasets, each dataset includes a task definition and a set of fixed positive and negative demonstrations. We follow the previous research [13; 26] and examine two settings: only using the task definition as the input to the hypernetwork (‘Def’), and using the definition along with two few-shot positive examples (‘Def + 2 Pos’). We only use the English tasks in the dataset and the model’s generation is evaluated on a set of 119 unseen tasks using ROUGE-L.

P3 composed of 62 task datasets, the T0 model is trained with these tasks divided into meta-training and meta-test sets. The format of the prompts takes into consideration 0-shot reasoning and typically includes instructions or possible answer options. We follow the precedent work [40] by using the T0 training subset 36 tasks to train our model. The evaluation is conducted based on the accuracy scores of multiple-choice questions for unseen 11 tasks in the meta-test set (MTest11).

	Pre-	Instr.	Low Infer.	Instr.	Unseen
Method	Train	Fus.	Cost	Learning	Task
Simple FT	✗	✔	✗	✗	✗
T0 [29] / Tk-Instruct [36]	✗	✔	✗	✗	✔✔✔
Hypter [41]	✗	✗	✔	✗	✔
HyperTuning [26]	✔	✗	✔	✗	✔
HINT [13]	✔	✗	✔	✗	✔✔
TAGI (Ours)	✔	✔	✔	✔	✔✔✔

4.2 Baselines

We compare the characteristics of TAGI against eight primary groups of baselines (as shown in Table1): 1) No FT: models without finetuning. 2) HyperTuning [26]: models that use hypernetwork to convert demonstrations into adapters without instruction fusion. 3) Hypter [41]: models based on hypernetwork do not use pretraining. 4) HINT [13]: models pretrain hypernetwork and concat instruction. 5) T0 and Tk-Instruct: strong baselines fully finetuned on P3 and SNI respectively with instruction concatenated. 6) Full FT: models fineuned on target tasks. 7) Decoder-only model: decoder-only models fully finetuned like GPT-2 [27] and OPT [42]. 8) FiD-ICL [40]: ICL method use encoder intermediate fusion.

4.3 Implementations

We limit our scope to encoder-decoder models for our experiments³³3We have discussed in detail the encoder-decoder and decoder-only models in B.1.. We use T5-LM-Adapt⁴⁴4https://huggingface.co/google/t5-xl-lm-adapt and T0 [29] as initializations in our experiments. The two model groups have the same architectural framework but differ in weight; T0 uses T5-LM-Adapt for initialization and undergoes multi-task training on the P3 meta-training set. For SNI, only T5-LM-Adapt is considered, and three different sizes are tested: Base (250M), XL (3B), and XXL (11B), with the teacher model being TK-Instruct [36]. For P3, we experimented with two sets of models of three different sizes: Base (250M), Large (800M), and XL (3B) with the only template as input, while the teacher model being FiD-ICL [40] with 16-shot examples. The A.4 contains more implementation details and experimental settings.

4.4 Main Results

Super-Natural Instructions. We report the performance and inference costs of TAGI models and baselines in subsection4.4. Our analysis and findings yield several key insights:

$\bullet$ Firstly, methods lacking finetuning exhibit subpar performance. As shown in the first row of the table, the performance of No FT is significantly lower than other baseline methods by approximately 30 points (except for Hypter), which underscores the critical role of inductive bias, introduced during meta-training, in enhancing the model’s instructional adherence and cross-task generalization.

	Def (Zero-shot)			Def + 2 Pos. (Few-shot)			Avg. Rel.
Method	Base (250M)	XL (3B)	XXL (11B)	Base (250M)	XL (3B)	XXL (11B)	FLOPs
No FT	8.8	14.3	26.2	9.4	13.6	30.5	$\times$ 1.0
Tk-Instruct^†	35.3	48.0	53.6	42.1	54.0	62.0	$\times$ 1.0
\hdashline # Decoder-only model
GPT-2 XL (1.5B)^∗	-	38.2	-	-	45.3	-	$\times$ 0.33
OPT (13B)^∗	-	-	44.8	-	-	51.5	$\times$ 0.36
\hdashline # Hypernetwork-based model
Hypter^∗	12.1	16.8	15.5	10.6	14.2	13.4	$\times$ 0.35
HyperTuning^†	-	38.9	-	-	48.6	-	$\times$ 0.34
HINT^∗	33.3	47.2	51.1	41.8	53.2	56.4	$\times$ 0.37
TAGI (Ours)	35.3	48.4	52.3 ^‡	42.5	56.3	58.4 ^‡	$\times$ 0.39

	T5-LM			T0			Avg. Rel.
Method	Base (250M)	Large (800M)	XL (3B)	Base (250M)	Large (800M)	XL (3B)	Infer. Time
# MTest11 Avg.
Zero-shot	43.9	41.5	42.6	49.1	52.4	57.6	$\times$ 1.0
Full FT	44.6	45.5	47.2	51.9	56.6	61.4	$\times$ 1.0
Metatrain ^♡	44.1	52.4	53.1	50.1	52.4	56.8	$\times$ 1.0
\hdashline # ICL-based method
Concat-ICL^α	44.2	47.6	-	48.6	53.2	-	$\times$ 4.1
FiD-ICL^α	47.0	55.2	60.0	51.0	53.4	58.2	$\times$ 1.9
Ensemble-ICL^α	44.6	54.5	52.6	49.9	53.7	57.7	$\times$ 13.2
\hdashline # Hypernetwork-based model
Hypter^∗	-	-	-	-	-	56.2	-
HINT^∗	-	-	-	-	-	60.3	-
TAGI (Ours)	45.6	54.7	58.9	50.8	53.8	58.8	$\times$ 0.88
# HyperT5 Avg. (Without SCloze dataset)
FiD-ICL^α	46.9	55.8	60.6	51.7	53.9	58.5	$\times$ 1.9
HyperTuning^†	-	54.6	59.6	-	-	-	-
TAGI (Ours)	46.7	56.0	59.8	51.7	54.6	59.2	$\times$ 0.88

Method	Def	Def + 2Pos.	P3
TK-Instruct	48.0	54.0	-
TK-Instruct-LoRA	47.5	54.6	-
TK-Instruct-Prefix	42.6	54.2	-
\hdashlineHypertuning	38.9	48.6	59.6
HINT	47.2	53.2	60.3
TAGI	48.4	56.3	60.6
Ablation Study
w/o pretraining	47.1	55.6	58.3
w/o Instr. Fus.	35.1	40.6	44.2
w/o $\mathcal{L}_{\text{ce}}$	47.6	55.4	59.8
w/o $\mathcal{L}_{\text{kl}}$	45.7	53.9	57.3
w/o $\mathcal{L}_{\text{ins}}$	47.5	55.2	59.4
w/o Hypernetwork	43.8	50.7	-

Dataset	Examples per Task	Train	Test
Super-Natural Instructions	100	75,417	11,810
P3	-	90,897,454	2,940,068
P3 (Sampling)	1000	290,000	2,940,068

			Finetuning
			SNI			P3
	LoRA Tuning	Pretraining	Base (250M)	XL(3B)	XXL (11B)	Base (250M)	Large (800M)	XL(3B)
Max Input Len	1024	1024	1024	1024	1024	512	512	512
Max Output Len	128	-	128	128	128	64	64	64
Optimizer	adamw	adafactor	adamw	adamw	adamw	adamw	adamw	adamw
Learning Rate	1e-4	1e-3	1e-4	5e-5	5e-5	1e-4	1e-4	5e-5
precision	bf16	float32	bf16	bf16	bf16	bf16	bf16	bf16
# Training Steps	10000	50000	20000	20000	20000	20000	20000	20000
# Warmup Steps	-	-	# 2% of total training steps
Batch Size	8	8	8	2	1	8	4	2
Gradient Accumulation	2	1	2	4	2	2	4	4
LoRA Rank	# 32

Task Adapters Generation from Instructions (2024)

Abstract

1 Introduction

2 Related Work