I recommend the books by Joe Suzuki https://bayesnet.org/books/ it teaches all the math and no libray/framework is used.

edit: the readers page is not up to date the last books are available.