The original Huddersfield Library and Art Gallery was founded in 1898 by the County Borough of Huddersfield. It initially held exhibitions of loans from local collectors and the work of members of the Huddersfield Art Society (formed in 1890).

In 1940 a purpose built library and art gallery was completed, designed by the architect E.H. Ashburner. The building opened in the midst of World War II so the floor set aside for the gallery was instead used as an emergency hospital. It was not until 1943 that the gallery opened in this building. During the war years, the gallery hosted a number of small manageable exhibitions but it wasn't until after the war that the gallery was officially launched. The grand original opening exhibition which was scheduled to coincide with the opening of the building finally took place in 1946. Titled 'Two Hundred Years of British Art' it brought together works from public and private collections and was a comprehensive survey of British Art with works by Turner, Constable, Stubbs, Hogarth and Gainsborough. The exhibition was opened by Kenneth Clark, the art historian and director of the National Gallery.

Up until 2020, the grand building was home to the Kirklees art collections which was formed in 1974 as a result of local government reorganisation, which brought together the museum and art gallery collections of Huddersfield, Dewsbury, Batley and a number of small authorities and town councils that constituted Kirklees Metropolitan Council.