Hugging Face's logo

legraphista
/
unsloth-Phi-3.5-mini-instruct-IMat-GGUF

Text Generation
GGUF
Transformers
multilingual
unsloth
phi3
phi
quantized
GGUF
quantization
imat
imatrix
static
16bit
8bit
6bit
5bit
4bit
3bit
2bit
1bit
Model card Files Community
legraphista commited on
Commit
49fbd84
•
1 Parent(s): d675927

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +162 -0
imatrix.log ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ llama_model_loader: loaded meta data with 35 key-value pairs and 291 tensors from unsloth-Phi-3.5-mini-instruct-IMat-GGUF/unsloth-Phi-3.5-mini-instruct.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
2
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
3
+ llama_model_loader: - kv 0: general.architecture str = llama
4
+ llama_model_loader: - kv 1: general.type str = model
5
+ llama_model_loader: - kv 2: general.name str = Unsloth Phi 3.5 Mini Instruct
6
+ llama_model_loader: - kv 3: general.finetune str = instruct
7
+ llama_model_loader: - kv 4: general.basename str = unsloth-Phi-3.5
8
+ llama_model_loader: - kv 5: general.size_label str = mini
9
+ llama_model_loader: - kv 6: general.license str = mit
10
+ llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/microsoft/Phi-...
11
+ llama_model_loader: - kv 8: general.tags arr[str,4] = ["unsloth", "transformers", "phi3", "...
12
+ llama_model_loader: - kv 9: general.languages arr[str,1] = ["multilingual"]
13
+ llama_model_loader: - kv 10: llama.block_count u32 = 32
14
+ llama_model_loader: - kv 11: llama.context_length u32 = 131072
15
+ llama_model_loader: - kv 12: llama.embedding_length u32 = 3072
16
+ llama_model_loader: - kv 13: llama.feed_forward_length u32 = 8192
17
+ llama_model_loader: - kv 14: llama.attention.head_count u32 = 32
18
+ llama_model_loader: - kv 15: llama.attention.head_count_kv u32 = 32
19
+ llama_model_loader: - kv 16: llama.rope.freq_base f32 = 10000.000000
20
+ llama_model_loader: - kv 17: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
21
+ llama_model_loader: - kv 18: general.file_type u32 = 7
22
+ llama_model_loader: - kv 19: llama.vocab_size u32 = 32064
23
+ llama_model_loader: - kv 20: llama.rope.dimension_count u32 = 96
24
+ llama_model_loader: - kv 21: tokenizer.ggml.add_space_prefix bool = false
25
+ llama_model_loader: - kv 22: tokenizer.ggml.model str = llama
26
+ llama_model_loader: - kv 23: tokenizer.ggml.pre str = default
27
+ llama_model_loader: - kv 24: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
28
+ llama_model_loader: - kv 25: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00...
29
+ llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
30
+ llama_model_loader: - kv 27: tokenizer.ggml.bos_token_id u32 = 1
31
+ llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 32000
32
+ llama_model_loader: - kv 29: tokenizer.ggml.unknown_token_id u32 = 0
33
+ llama_model_loader: - kv 30: tokenizer.ggml.padding_token_id u32 = 32009
34
+ llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
35
+ llama_model_loader: - kv 32: tokenizer.ggml.add_eos_token bool = false
36
+ llama_model_loader: - kv 33: tokenizer.chat_template str = {% for message in messages %}{% if me...
37
+ llama_model_loader: - kv 34: general.quantization_version u32 = 2
38
+ llama_model_loader: - type f32: 65 tensors
39
+ llama_model_loader: - type q8_0: 226 tensors
40
+ llm_load_vocab: special tokens cache size = 14
41
+ llm_load_vocab: token to piece cache size = 0.1685 MB
42
+ llm_load_print_meta: format = GGUF V3 (latest)
43
+ llm_load_print_meta: arch = llama
44
+ llm_load_print_meta: vocab type = SPM
45
+ llm_load_print_meta: n_vocab = 32064
46
+ llm_load_print_meta: n_merges = 0
47
+ llm_load_print_meta: vocab_only = 0
48
+ llm_load_print_meta: n_ctx_train = 131072
49
+ llm_load_print_meta: n_embd = 3072
50
+ llm_load_print_meta: n_layer = 32
51
+ llm_load_print_meta: n_head = 32
52
+ llm_load_print_meta: n_head_kv = 32
53
+ llm_load_print_meta: n_rot = 96
54
+ llm_load_print_meta: n_swa = 0
55
+ llm_load_print_meta: n_embd_head_k = 96
56
+ llm_load_print_meta: n_embd_head_v = 96
57
+ llm_load_print_meta: n_gqa = 1
58
+ llm_load_print_meta: n_embd_k_gqa = 3072
59
+ llm_load_print_meta: n_embd_v_gqa = 3072
60
+ llm_load_print_meta: f_norm_eps = 0.0e+00
61
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-05
62
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
63
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
64
+ llm_load_print_meta: f_logit_scale = 0.0e+00
65
+ llm_load_print_meta: n_ff = 8192
66
+ llm_load_print_meta: n_expert = 0
67
+ llm_load_print_meta: n_expert_used = 0
68
+ llm_load_print_meta: causal attn = 1
69
+ llm_load_print_meta: pooling type = 0
70
+ llm_load_print_meta: rope type = 0
71
+ llm_load_print_meta: rope scaling = linear
72
+ llm_load_print_meta: freq_base_train = 10000.0
73
+ llm_load_print_meta: freq_scale_train = 1
74
+ llm_load_print_meta: n_ctx_orig_yarn = 131072
75
+ llm_load_print_meta: rope_finetuned = unknown
76
+ llm_load_print_meta: ssm_d_conv = 0
77
+ llm_load_print_meta: ssm_d_inner = 0
78
+ llm_load_print_meta: ssm_d_state = 0
79
+ llm_load_print_meta: ssm_dt_rank = 0
80
+ llm_load_print_meta: ssm_dt_b_c_rms = 0
81
+ llm_load_print_meta: model type = 7B
82
+ llm_load_print_meta: model ftype = Q8_0
83
+ llm_load_print_meta: model params = 3.82 B
84
+ llm_load_print_meta: model size = 3.78 GiB (8.50 BPW)
85
+ llm_load_print_meta: general.name = Unsloth Phi 3.5 Mini Instruct
86
+ llm_load_print_meta: BOS token = 1 '<s>'
87
+ llm_load_print_meta: EOS token = 32000 '<|endoftext|>'
88
+ llm_load_print_meta: UNK token = 0 '<unk>'
89
+ llm_load_print_meta: PAD token = 32009 '<|placeholder6|>'
90
+ llm_load_print_meta: LF token = 13 '<0x0A>'
91
+ llm_load_print_meta: EOT token = 32007 '<|end|>'
92
+ llm_load_print_meta: max token length = 48
93
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
94
+ ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
95
+ ggml_cuda_init: found 1 CUDA devices:
96
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
97
+ llm_load_tensors: ggml ctx size = 0.27 MiB
98
+ llm_load_tensors: offloading 32 repeating layers to GPU
99
+ llm_load_tensors: offloading non-repeating layers to GPU
100
+ llm_load_tensors: offloaded 33/33 layers to GPU
101
+ llm_load_tensors: CPU buffer size = 99.81 MiB
102
+ llm_load_tensors: CUDA0 buffer size = 3772.57 MiB
103
+ .................................................................................................
104
+ llama_new_context_with_model: n_ctx = 512
105
+ llama_new_context_with_model: n_batch = 512
106
+ llama_new_context_with_model: n_ubatch = 512
107
+ llama_new_context_with_model: flash_attn = 0
108
+ llama_new_context_with_model: freq_base = 10000.0
109
+ llama_new_context_with_model: freq_scale = 1
110
+ llama_kv_cache_init: CUDA0 KV buffer size = 192.00 MiB
111
+ llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB
112
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB
113
+ llama_new_context_with_model: CUDA0 compute buffer size = 68.62 MiB
114
+ llama_new_context_with_model: CUDA_Host compute buffer size = 7.01 MiB
115
+ llama_new_context_with_model: graph nodes = 1030
116
+ llama_new_context_with_model: graph splits = 2
117
+
118
+ system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
119
+ compute_imatrix: tokenizing the input ..
120
+ compute_imatrix: tokenization took 143.583 ms
121
+ compute_imatrix: computing over 151 chunks with batch_size 512
122
+ compute_imatrix: 0.42 seconds per pass - ETA 1.05 minutes
123
+ [1]5.6455,[2]4.2496,[3]4.2246,[4]4.7837,[5]5.2227,[6]5.3658,[7]4.8447,[8]5.3686,[9]5.5685,
124
+ save_imatrix: stored collected data after 10 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
125
+ [10]5.9472,[11]5.9353,[12]5.4576,[13]5.3625,[14]5.6370,[15]6.0624,[16]6.1669,[17]6.4863,[18]6.6621,[19]6.8056,
126
+ save_imatrix: stored collected data after 20 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
127
+ [20]6.9293,[21]7.1482,[22]6.8526,[23]6.5285,[24]6.5929,[25]6.6662,[26]6.5827,[27]6.4753,[28]6.5536,[29]6.7352,
128
+ save_imatrix: stored collected data after 30 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
129
+ [30]6.8579,[31]6.8264,[32]6.9704,[33]7.0894,[34]7.2737,[35]7.2872,[36]7.2740,[37]6.9656,[38]6.7646,[39]6.6571,
130
+ save_imatrix: stored collected data after 40 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
131
+ [40]6.5504,[41]6.4638,[42]6.3819,[43]6.2510,[44]6.1741,[45]6.0896,[46]6.0434,[47]6.0518,[48]6.1210,[49]6.2249,
132
+ save_imatrix: stored collected data after 50 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
133
+ [50]6.2443,[51]6.4326,[52]6.6097,[53]6.8114,[54]7.0049,[55]7.1141,[56]7.0518,[57]6.9795,[58]6.9947,[59]7.0529,
134
+ save_imatrix: stored collected data after 60 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
135
+ [60]7.1530,[61]7.0435,[62]7.0612,[63]7.1083,[64]7.1889,[65]7.2617,[66]7.2995,[67]7.3482,[68]7.4029,[69]7.3956,
136
+ save_imatrix: stored collected data after 70 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
137
+ [70]7.4210,[71]7.4304,[72]7.4420,[73]7.3864,[74]7.3039,[75]7.2846,[76]7.3311,[77]7.3356,[78]7.2983,[79]7.2867,
138
+ save_imatrix: stored collected data after 80 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
139
+ [80]7.2890,[81]7.2540,[82]7.2330,[83]7.2053,[84]7.2096,[85]7.2169,[86]7.2051,[87]7.2006,[88]7.1903,[89]7.1772,
140
+ save_imatrix: stored collected data after 90 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
141
+ [90]7.1623,[91]7.1720,[92]7.1281,[93]7.1245,[94]7.0971,[95]7.0535,[96]7.0674,[97]7.0479,[98]7.0504,[99]7.0249,
142
+ save_imatrix: stored collected data after 100 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
143
+ [100]7.0132,[101]7.0250,[102]6.9877,[103]6.9490,[104]6.9416,[105]6.9636,[106]6.9712,[107]6.9971,[108]7.0300,[109]6.9885,
144
+ save_imatrix: stored collected data after 110 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
145
+ [110]6.9476,[111]6.9077,[112]6.8681,[113]6.8215,[114]6.7730,[115]6.7377,[116]6.6973,[117]6.6657,[118]6.6775,[119]6.6842,
146
+ save_imatrix: stored collected data after 120 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
147
+ [120]6.7383,[121]6.7896,[122]6.8503,[123]6.9010,[124]6.9832,[125]7.0567,[126]7.0667,[127]7.0719,[128]7.0068,[129]7.0001,
148
+ save_imatrix: stored collected data after 130 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
149
+ [130]6.9700,[131]6.9500,[132]6.9093,[133]6.8680,[134]6.8811,[135]6.9009,[136]6.8981,[137]6.8963,[138]6.9030,[139]6.9162,
150
+ save_imatrix: stored collected data after 140 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
151
+ [140]6.9331,[141]6.9349,[142]6.9355,[143]6.9334,[144]6.9116,[145]6.9291,[146]6.9632,[147]7.0090,[148]7.0513,[149]7.0970,
152
+ save_imatrix: stored collected data after 150 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
153
+ [150]7.1404,[151]7.1870,
154
+ save_imatrix: stored collected data after 151 chunks in unsloth-Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
155
+
156
+ llama_print_timings: load time = 1420.67 ms
157
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
158
+ llama_print_timings: prompt eval time = 55358.52 ms / 77312 tokens ( 0.72 ms per token, 1396.57 tokens per second)
159
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
160
+ llama_print_timings: total time = 56810.47 ms / 77313 tokens
161
+
162
+ Final estimate: PPL = 7.1870 +/- 0.09220