The traders, the immigrants and the inhabitants.

From very early times, India carried on a flourishing trade with the Middle East and the Western countries, particularly with Antioch, Alexandria and Rome by both land and sea.
The earliest people to visit India from the Middle East were Arabs as early as 3000 BC. The ancient Tamil Literatures mention about Yavanas (fair people), the Greeks and Romans who had been given permission to build fortified towns called Yavanappatinam or Yavanacherry.
Even before 1000 BC there was a large number of Jews in the region now called Kerala who used to trade with Syria, Palestine and other Middle Eastern countries. When the Roman empire came to power they found sea routes to India from Syria and Palestine. They bought commodities like pepper, ginger, cinnamon, cardamom and sandalwood in return for gold, silver, copper, bronze and perfumes.
Hippaulus, a Greek sailor found a sea route to India in 600BC through Red Sea.
Many Jews settled in Kerala during the period of Nebuchadnezzar in 6th century BC in Yavanacherry. The Jews might have settled in the Chera city of Kodungalloor and the Pandiyan city of Nelcynda.
The Jain Pandiyan Kings used to rule most of the Kerala prior to 300 BC when the Chera kingdom was founded by the immigrants (Chera, Vellian and Puraiyans). Pandiyans were well known to Greek historian Tolemy while Pliny Elder visited the Chera capital of Kodungalloor in 24 AD.
The Pandiyan navy found sea roots from Syria and Persia to China.

There is a mention about Essenes (a religious group) in "Manimekhalai" (tamil verse epic of 2nd century) who could have been converted Jewish Christians. Brahmins were virtually non extinct in Sangha Literature during the Christian period.
The Christians who had arrived from the Middle East in the 3rd century AD and the Muslims who arrived in the 8th century AD were generally traders and were not involved in this social segregation and generally kept aloof from the ambit of caste politics. The Jews who arrived Kerala in the early years of the Christian era were given privileges to trade and thus became an influential part of the melting pot of Kerala's population.