How might I handle event-sourcing?

I've never been happy with the way I've implemented certain kinds of features:

• Notifications
• Some kinds of reporting
• Activity Feeds

Then, I saw two things:

1. The "bucket/recording" pattern inside https://www.youtube.com/watch?v=m1jOWu7woKM
2. This post on event sourcing by kickstarter: https://kickstarter.engineering/event-sourcing-made-simple-4a2625113224

This is a really useful pattern that forms the basis of the way I'd like to build rails apps whose main functions are informatoin systems with hard needs for reporting, notifications, and auditing. (I'm also gonna use the definitions/words from above as my language for this thread). Now, I need to understand the particulars of how to do this. It seems like there are basically two approaches:

• ditching rails conventions, and stitching together a bunch of home-grown objects (see the kickstarter way)
• deeply embracing rails and implementing an event-sourcing approach inside of it (the bc3 way)

Of the two, I'm a lot more interested in the deeply embracing rails approach because I think that will stand the test of time. To do this, the goal is to build a simple little app: "Stuff Tracker." Basically, it's purpose is to store a list of items of things in my house I'd like to get rid of, store basic info (name, note, where stored), store images about them (to expose a list of things to sell), and move them through a simple state machine to ultimately sell/discard/store each thing. This way I can easily begin tracking all of the stuff in my house I need to get rid of, and easily expose lists of various types to other people (can I sell this thing? would you like to buy anything on this list of items that's "for sale?" etc.) to arrive at a decision on what to do with it.

Purpose of this thread

I left the notebook I'd stored all my notes in about my thoughts here, and recently the number of computers and environments I code on exploded. To remedy this, and rebuild my notes, I'm gonna store the progression of thought on here. This would have an added benefit: perspective. So here's my ask: if you're reading through this and see a better way to do things, please either comment and/or pr! I'm not exactly sure what I'm doing, so different POV's would be really helpful.

I'm not interested in a 1:1 mapping of "the event sourcing pattern" into rails. I'm looking for an "event-sourcing-like" implementation usable across projects and that fits into the rails way. this means embracing:

• concerns
• callbacks
• framing things in terms :resources the way they'd get defined in routes (will deal with graphql later)
• NOT dealing with problems of distributed systems

Getting Started

Starting here: https://github.com/ZempTime/stuff-tracker/blob/master/app/controllers/items_controller.rb#L10

When creating an event, a couple things jump to mind we'd need to handle:

• wrap all associated mutations to the db in a transaction
• store in a specific context (so for multitenancy, think a tenant. this notion is likely expanded across the notification paths of various projects/teams and stored in the notion of a "bucket")
• we want to store the event, then find_or_create the associated aggregate
• where does the calculator live? (Is the event the model itself?)
• reacting to sync and async things we want to happen (basically only sync thing for now is creating the item)
• dealing with nested records (won't have to handle immediately, but soon). So I think this is what the "parent_recording" is in the video. Basically storing related sequences of a request in a little graph so you can transact a bunch of parts of what forms a single action in the system

I've got a little time remaining today. Immediate questions I want to get a basic implementation of:

• where do I store validations? (I'm thinking for now just on the model)
• how do I determine a top-level context (or bucket) that this event gets tracked inside/against? (think I might just make a bucket model, too, and defer the decision of scoping to that bucket)
• here's what my "events" currently look like: https://github.com/ZempTime/stuff-tracker/blob/master/db/migrate/20181201212545_create_events.rb should I tie the aggregate to a bucket, or the event to a bucket and an aggregate?
• testing

I think I'm gonna blindly follow the idea of "bucket" and the idea that if a thing is "recordable" it can consist of a discrete set of events (meaning, if a thing is recordable then that means its an aggregate). "Bucketable" means it can behave as a bucket. Ultimately I think these should be real domain concepts (like tenant, or project), but I'm gonna go ahead and make a literal bucket and merge that into some other sort of differentiation later.

I haven't made enough progress to commit on, but it seems as though:

• I'll need the concept of whatever my bucket is. So a Place model.
• The Event model is the right place to manage the create action via callbacks. I'm curious about using conventional event type names, and naming convention (is it item_created or item_create? are there existing parts of rails I can leverage to sorta handle/automatically set these calculations for me?)

Seems like the Event model will need to:

before_validation :set_aggregate # pull in validations from appropriate model
after_validation :track_event # persist this event if validations pass
after_create :apply_event # apply the contents of this event to the appropriate aggregate. in the case of creating an item, find_or_create

So the question becomes, where does the calculation logic for this event live. Kickstarter says, in the event definition. I think maybe I can just defer to Item.create for right now, though

So it appears this is the big rub:

• From what I can tell from bc3, they create events after the fact about things that have already happened
• Kickstarter/the pattern itself definitely prescribe creating the event first, then applying it

As long as validations fire off for things that aren't generated by the system/world, all should be fine... right?

I'd really like the best of both worlds. Don't wanna jump whole hog out of using typical rails methods. But I think there's value in the event being created first, then applied because that way things can be recalculated, retried, etc.

So I think what makes the most sense is layering in a new custom active record primitive into the system, much like "create" or "update" - "track."

Then as for where calculators for events live, they'll live on whatever model makes sense. The model itself for typical form interactions, activemodel-backed classes for form interactions that make sense, etc.

This slightly shifts create and update.

• create -> essentially becomes find_or_create
• update -> doesn't simply accept a single set of updates. it needs to be able to accept many sets of updates all at once, consisting of at least one.

This means track becomes the method inside controllers, create and update need to be slightly wrapped, and everything is still forced to be phrased through whatever it is.

This solves for

• aggregates
• calculators
• events

Doesn't solve for reactors yet, but once there are a variety of events happening, I think this will take the form of some centralized monitoring layer, maybe able to just get popped inside ApplicationRecord.