It appears that the word panjibhai was coined most probably in the tradition of Indo-Pakistan subcontinent during 17th century. This word was also used in the 14th part of the old prayer. The word panjibhai means the hand shaking brother. It was generally used for those who served inside the prayer-hall. It became more common for those who served inside and outside the community during the Aga Khan Case of 1866. On that occasion, some eminent Ismailis had rendered valuable services in all affairs, the most prominent among them were Ismail Kherraj, Sharif Gangji, Mukhi Alarakhia Sumar, Kamadia Khaki Padamsi and their workers, who were known as the panjibhais. The Imam recognized them as Ismaili fidais and Ismaili soldiers. Henceforward, the group of the panjibhai continued to serve in different fields, and need also arose to form an association of brotherhood in the community. Thus, the existence of their club came into being. This was the first association of brotherhood. It was initially a group of enthusiastic persons, working for the religious, economical and social welfare of the Ismailis in Bombay. Later on, it had been recognized an official institution of the community when a bungalow had been hired at Repon Road, Bombay. When Imam Sultan Muhammad Shah returned from his European trip in 1899, the selected members sought permission for its establishment. The Imam named it The Khoja Panjibhai Club and himself became its life patron. The Club was later on shifted at Chopati Road, which was inaugurated by the Imam on May 28, 1900, whose first President was Jafar Rehmatullah with the Hon. Secretary, Ibrahim Muhammad Rawji. Before it, the Khoja Panjibhai Club had been founded in Zanzibar on September 9, 1899, whose first President was Mitha Jessa with the Hon. Secretary, Muhammad Saleh Kanji. In Jamnagar, Kutchh The Khoja Panjibhai Club existed on August 16, 1904, whose first President was Kamadia Haji Nazar Ali with the Hon. Secretary, Ghulam Hussain Ali Bhai. The Khoja Panjibhai Club also opened in Karachi on June 26, 1905, whose first President was Ghulam Hussain Varas Wali with the Hon. Secretary, Alidina Ali Muhammad.

Later, the Khoja Panjibhai Club along with other relevant institutions were merged in Bombay under the name of The Aga Khan Gymkhana in 1936. The same changes also took place in Zanzibar, then in Karachi in 1940.