I like the idea, but is it really necessary to start from scratch (and moreover, I'm not sure if ActivityPub is the best fit for this if you're advocating privacy)? In the Fediverse, Hubzilla already has a calendar and event system, and so does Friendica. Perhaps Diaspora as well, I'm not too familiar wit that one.