I just got back from a conference in Lyon, my third academic conference this year.
Most people think it's strange I attend (and sometimes speak at) academic conferences that aren't part of my job.
Some of the academics at this last conference, when they found out, also thought it was strange but they at least seemed excited for me.
One thing that quickly becomes clear: people are so used to wanting to know your affiliation and it can thoroughly confuse them when you don't have one.