In a large mixing bowl, cream butter and sugar until light and fluffy. Add egg yolks; beat until light. Sift together flour and baking powder; add cracker crumbs. Add alternately with coffee to creamed mixture, beating well until smooth. Stir in vanilla and nuts.
In another bowl, beat egg whites until stiff; fold into batter. Pour into two 8-in. round waxed paper-lined baking pans. Bake at 350° for 30-35 minutes. Cool completely.
For filling, prepare pudding with milk and coffee; chill. Fold in cream. Split each cake layer and spread with filling.