Agree with the caretaker priest not being too friendly.\r Don't know about him not liking caucasions, however, nothing seems to be happening there except for New Years day.
It's a place that must be seen and shared with people, even those who have no ties to Shinto will marvel at the simple beauty of the place. Beware the caretakers. They will remove you if you happen to be too white to be visting after hours. Please help to reopen the shrine. It should be open to the public and looked after by people who care about it more than they want a paycheck.