Rendered at 11:00:30 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
bananaflag 28 minutes ago [-]
This is the first thing I see in my life that really looks like proto-AGI, it deserves its name.
whizzter 3 hours ago [-]
Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.
awfm9 25 minutes ago [-]
What's the advantage of doing this, versus becoming good at context management and RAG? I always found trained knowledge unreliable, given that it is lossy by construction.
advael 3 hours ago [-]
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
comboy 12 minutes ago [-]
what character prediction rates are you getting on some unseen datasets?
volotat 9 minutes ago [-]
The held-out scores reported in the Readme IS the unseen dataset.
cpldcpu 3 hours ago [-]
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
volotat 2 hours ago [-]
The model is way too small and undertrained to make any generalization claims. I want to wait until it reads the whole corpus I gave and then test it on some simple established benchmarks to see how it will behave.
jacquesm 2 hours ago [-]
What kind of hardware are you using for training?
nm, I found it:
> RTX 3070 Laptop GPU with 8 GB
Super impressive.
hexley19 4 hours ago [-]
Seeing 'Mini-AGI' and '8GB VRAM' in the same sentence is a breath of fresh air. Maybe local AGI isn't so far-fetched.
skeledrew 3 hours ago [-]
Getting conceptually closer to how the human brain works. Looking forward to more of this.
volotat 3 hours ago [-]
I also like how it is very organic. It naturally grows and deletes unused elements, so in addition to traditional backprop there is also a natural selection happening in the background. Each new expert has 16 parents by the way, lol.
ilusion 2 hours ago [-]
Have you tested what it remembers from early in the stream after a shift in the topics thrown at it?
volotat 2 hours ago [-]
It interleaves random streams of 32K characters long each when reading the whole corpus, but each such stream reads continuously as you would expect. This is a necessary step to prevent just normal, not catastrophic, forgetting. I have not tested it in any other regimes yet with bigger or smaller windows. You can imagine a person that changes the activity from time to time, so I think it is justified. So there is not really "early in the stream".
What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.
nm, I found it:
> RTX 3070 Laptop GPU with 8 GB
Super impressive.
What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.