 Piotr Wilkin
|
aa8d6a21a3
Remove extra files cont.
|
vor 3 Monaten |
|
Piotr Wilkin
|
e9a98f2af9
Remove extra files
|
vor 3 Monaten |
|
Piotr Wilkin
|
22ee5a971b
Add gate_sigmoid to callback
|
vor 3 Monaten |
|
Piotr Wilkin
|
ce87b7d78e
Yup, it's NeoX
|
vor 3 Monaten |
|
Piotr Wilkin
|
df0b5bcf30
Proper order of attention operations
|
vor 3 Monaten |
|
Piotr Wilkin
|
54712b8664
Oh, forgot to commit
|
vor 3 Monaten |
|
Piotr Wilkin
|
17240eafc0
Order stuff around
|
vor 3 Monaten |
|
Piotr Wilkin
|
1579bcb202
What am I missing? :/
|
vor 3 Monaten |
|
Piotr Wilkin
|
0a9244acd0
The optimization worked even too well ;)
|
vor 3 Monaten |
|
Piotr Wilkin
|
8ddaf251ae
Fix some state regressions... still wip
|
vor 3 Monaten |
|
Piotr Wilkin
|
6942c85cf8
Oh, actually set n_tasks as well :P
|
vor 3 Monaten |
|
Piotr Wilkin
|
477c1616ad
Parallelize delta_net
|
vor 3 Monaten |
|
Piotr Wilkin
|
4ef6f337de
Proper multi-sequence convolution calculation, corrected (?) state management
|
vor 3 Monaten |
|
Piotr Wilkin
|
5f5e30007c
Dilution n_seqs -> 1
|
vor 3 Monaten |
|
Piotr Wilkin
|
eb0a15fc9b
n_tokens -> n_seq_tokens
|
vor 3 Monaten |
|
Piotr Wilkin
|
ee52fe36f3
Modify sanity check to handle hybrid models
|
vor 3 Monaten |
|
Piotr Wilkin
|
0dd6110fdc
v1.0
|
vor 3 Monaten |
|
Piotr Wilkin
|
adcbd9428f
Linear layer output convergence
|
vor 3 Monaten |
|
Piotr Wilkin
|
666fc0583d
Parity on delta!
|
vor 3 Monaten |
|
Piotr Wilkin
|
a2c7b6794e
Proper handling for n_tokens > GGML_DELTA_NET_CHUNK
|
vor 4 Monaten |
|
Piotr Wilkin
|
c1e46f62fa
Achieve pre-chunk-attention parity; remove most of the LLM generated crap
|
vor 4 Monaten |
|
Piotr Wilkin
|
c87e8d550c
Tensor preparation for delta_net complete
|
vor 4 Monaten |
|
Piotr Wilkin
|
7ec2df64a4
Added: tri, cumsum. Still a mess.
|
vor 4 Monaten |
|
Piotr Wilkin
|
6d0ad37cf4
Fix QKV extraction post-convolution
|
vor 4 Monaten |
|
Piotr Wilkin
|
845a3d7166
Convolution
|
vor 4 Monaten |
|
Piotr Wilkin
|
638057a29b
Transpose input for convolution
|
vor 4 Monaten |
|
Piotr Wilkin
|
835d389fc5
Fix BA views as well
|
vor 4 Monaten |
|
Piotr Wilkin
|
594c1f98ef
QKV splits done right
|
vor 4 Monaten |
|
Piotr Wilkin
|
dbd4d97cf2
Fix cb calls
|
vor 4 Monaten |
|
Piotr Wilkin
|
32dcee47ef
Some attempts to get the convolution input right.
|
vor 4 Monaten |
