@khaleelkazi's thread

Rafiki Update

Just launched a bunch of optimizations for Rafiki including memory optimization and more frequent indexing of threads

We also added a watchdog feature so that he can monitor his hardware usage and ensure there isn't downtime when indexing new threads

For context: when Rafiki indexes the latest threads, his hardware kicks into overdrive (it is expensive to train in terms of ram usage). He processes the threads and will keep replies in a backup queue while he is doing it now (as opposed to before where he tries to reply to people + index threads + use tool calls)

This new update allows him to gracefully pause everything, index new threads and then restart. This allows him to focus all his hardware on training for ~60 seconds. During this 60 seconds, if a user asks Rafiki a question, Rafiki saves the question and replies after his 60 second index + reboot phase is complete

Vastly more robust to our system before + more frequent training