Andy was born in Plauen, Germany in 1924 but spent his childhood in Falkenstein. When the Nazis came to power, life for Andy became difficult. Andy was the only Jewish boy in his class and he experienced humiliation and isolation.
The day after Kristallnacht, the night of 9 November, 1938, when the Nazis arrested thousands of Jewish men and vandalised Jewish shops and synagogues, Andy and his father were arrested. Shortly after Andy was sent home but his father was sent to Buchenwald. He was released six weeks later – on condition that the Factor family leave Germany immediately. They fled to Paris until December 1939 when they secured permits to travel to Australia.