The important thing to remember about strict genre forms like yakuza-eiga is that these films are not necessarily individual works of art but instead variations on a complex tacit social metaphor, a secret agreement between the artists and the audiences of a certain period.
When massive social forces are in flux, rigid genre forms often arise to help individuals make the transition. Americans created the Western to help codify a morality of the frontier; they created a gangster film to cope with the new social forces of the city.
If the original social metaphor is valid, the resulting genre will long outlive the individual artists who created it -- it may even outlive the times which evolved it. In the present personality-oriented culture, rigid genre forms are the closest thing we have to a popular "art without names."
When a new genre comes into being, one immediately suspects that its causes run far deeper than the imagination of a few astute artists and businessmen. The whole social fabric of a culture has been torn, and a new metaphor has arisen to help mend it.
The yakuza-eiga is a popular social contract between the artists and the audiences of Japan to reevaluate and restructure these traditional virtues. The Samurai Film was no longer serving its intermediary function; new characters, themes and conventions had to be created. Just as early twentieth century Americans needed the Western, contemporary Japanese need a genre which can serve as a moral battleground -- a genre on which the traditional virtues can fight to the death.
Genre and art are not mutually exclusive. The western was a genre, a knight-errant tale of the frontier, and Sergio Leone raised it to art, as did TheAwesome movie Unforgiven and the tv series Deadwood.