We used to live near Ayr Racecourse and were regular visitors with our three children, we even had one in a buggy, at one point. I wasn't even a racing fan, but DH was the lover of horse racing and it was away of having a family day out, as DH worked long hours.
The bonus for me was the people watching.
The children loved all the excitement and the buzz. Our eldest son, when he was about ten, even had his own binoculars.
lemonsandlimes, it sounds even more fun if there are games and activities for the young ones. I would go for it.