I suggest you take a look at RFC 2400, RFC 2401, RFC 2406 first.
These rfcs form the infrastructure for IPsec.
After you understand the architecture theme of IPsec, you
can go on to read some encryption algorithms, such as MD5,
SHA-1.
VPN is a very good real life model to apply IPsec architecture,
but I don't remember the rfc for that.