Armfield was born in Sanderstead, near Croydon, Surrey, but grew up in Somerset. He studied design and the history of architecture at the Royal West of England College of Art in Bristol, then from 1934 to 1939 was Assistant Art Director at Ealing Film Studios. A Quaker pacifist, he returned on the outbreak of World War II to the home of his parents in Axbridge, Somerset.

After a nervous breakdown, he moved to Polperro in 1942. He and his partner, George Bray, lived high above The Warren in their home, 'Tycara'. By 1945 he felt confident enough to take up painting as a full time professional career. His cousin, the painter and writer Maxwell Ashby Armfield, taught him the tempera technique, which Stuart developed and in 1958 published a book on the subject in the Artists Handbook series.

Armfield's principal subject matter was still life. Unusually, the backdrops were generally an infinite space, evoking sea or sky in such a way as to suggest the influence of surrealism. Reviews of his work on the exhibitions of the East Cornwall Society of Artists and Plymouth Arts Club in the 1940s and 1950s praised the skill of his tempera technique. The paintings included in his first STISA exhibition in 1950 attracted much favourable attention. In the summer of 1956 his solo show at Plymouth Arts Centre (opened by fellow Polperro resident Jack MERRIOTT) was highly praised.

In 1965 he opened a gallery on the quay in Looe. Between 1958 and 1970 he shot a series of films in and around Looe, some of which have been uploaded onto YouTube by Cornish Voices. In 1966 The Cornish Review was relaunched, and from then until 1970 Armfield was the portrait artist for the publication. In 1971 he held an exhibition at The Orangery, Saltram House, which was opened by Lady Trehane. In the same year he was involved in the renovation of the Jubilee Inn, Pelynt, creating an original design for its glazed circular staircase. The 1950s and 1960s marked the highest point of his career, with several exhibitions in the USA establishing a following there.

Plymouth Art Gallery held a major retrospective of his work in 1977. Two years later he moved there, converting a dilapidated building on the Barbican into a new studio and gallery.