Originally founded in 1893 as the Saddle River Township Building and Loan Association, Spencer's history and values are deeply rooted to the history and values of the neighborhoods in New Jersey. The Association's original mission was to assist members acquire and finance homes by approving small and medium sized first mortgages. In the first twenty years, from 1894 to 1914, approximately 40 home mortgage loans were approved. The average size mortgage loan at this time was $935. As was the case for many financial institutions, Saddle River Township Building and Loan Association struggled during the Great Depression. In order to remain solvent, it liquidated its assets in 1939, and reincorporated as Spencer Building and Loan Association. In 1950, the board decided to take advantage of the Federal Savings and Loan Insurance Corporation and changed its charter to the Spencer Savings and Loan Association.

The Bank's original headquarters was on Passaic Street in Garfield. The first branch office was opened in Lodi (1955), quickly followed by Outwater Lane in Garfield (1956). The Wallington branch was opened in 1960 and Saddle Brook in 1961. In 1976, the Bank expanded into Clifton followed in 1977 by the Lyndhurst office. Four years later, in 1980, a second office was opened in Saddle Brook. The River Drive, Garfield branch was added in 1985. In 1988, Spencer Savings Bank acquired the Elmora Savings and Loan Association. This institution was comprised of four offices, two of which Spencer closed. The Magie branch in Union and Bayway branch in Elizabeth were the net result of this acquisition. In 1994, the Bank continued to expand its franchise with an office in Nutley. At that point in time the Bank had 12 offices in four counties - Bergen, Essex, Passaic and Union. The Bank relocated its corporate headquarters to Elmwood Park in 2002.

Since this time, Spencer has opened seven new branches in Cranford, Garwood, North Caldwell, Roselle, Clifton (2nd branch office), Elmwood Park and Wayne.