Neighbour dilemma

This is difficult without seeing perhaps even a small drawing of your garden to advise on what to do in your t block or deaden some of the noise. I'm you won't stop the whistling by asking so just wait a while and he will probably get tired trying to annoy and find something else to do. My daughter lives next to a woman who sings along to the radio in a very high pitched voice. She just put on her own music, loud!

I had this problem a couple of years ago although mine was young children screaming in the garden from 8am until evening on any day that was dry.
It was incessant and sometimes including visiting children .
I used to go indoors until I had the bright idea of buying noise cancelling headphones.
A drastic solution I know but I could sit out in my garden to just read or enjoy the weather.
It had lessened now the children are older and not so loud.
You also have the advantage of winter coming when presumably neither if you will be out so much.
Next year he may have stopped his whistling and taken up something else, I hope for your sake it isn’t a musical instrument.
Seriously though I do understand, I was driven inside by the noise until the headphones arrived.

One side of me says he is arranging his garden features and whistling just to annoy you. Noise is a stressor whether it's factory level or whistling.
You don't mention a partner or children. If he is on his own then maybe he isn't very tuned in to consider others.
You have a tricky situation here because as you say he can erect sheds etc wherever he wants to on his property within reason and whistling may be seen to those who don't have to listen to it constantly as a sign of being happy. To complain about any of this may make you look like a misery.
However, it is clear that your quality of life has been affected and that isn't fair. Are you able to rearrange your garden to put maximum distance between you and the fence?
As he has been hostile to you before I doubt he will comply with any request to stop whistling.
Headphones may be some sort of compromise but it will hinder conversation.
My elderly next door neighbour who I share a fence with has gone to live with his sister and sold his house to his nephew. Said nephew is very pleasant but he has converted his garage which forms part of our boundary into a man cave complete with flashing lights and quite loud music.
Both of us will be looking forward to a nippy autumn where our respective thorns will retire to their homes.