Now that you mention #XMPP.. at the time that https://openengiadina.net dropped #ActivityPub support for XMPP I was tempted to follow along. So many things are better documented in that open standard and the subsequent XEP's.
Yet also both XMPP and #Matrix are first and foremost instant messaging / chat protocols, and to me using them to build other abstractions on top of felt forced, plus being among the first and only ones to do so.